OpenX Crashing Server at 30,000 Impressions

sumohax0r

sumohax0r

what that be like?
Nov 1, 2009
740
41
0
127.0.0.1
Ok so I recently became in the need of a adserver for purchasing some media and after a little research I decided on OpenX because 1. It was free and 2. I could host it myself.

Ok so here is my problem and it would be GREAT if anyone could help.

I'm using a (dv) Dedicated Virtual server at MediaTemple - Spec:

GenuineIntel, Intel(R) Xeon(R)CPU L5420 @ 2.50GHz
1GB DDR3 RAM
CentOS
Plesk 8.6.0

-----------------------------------------------------

I have 2 apps running, Prosper202 and OpenX

-----------------------------------------------------

So I started testing small amounts of traffic to make sure this thing was not going to die, and well it did.

After impressions started coming in CPU went from 0.5% to around 90-98%.

RAM went from 600MB free to about 550MB free.

Plesk gave me warning errors saying that my "kmemsize" was in warning/black status.

Then the server would crash, Prosper202 goes down, then openx then I have to restart the entire server before anything works again.

I would think that this server would be able to handle 30,000 impressions, is there something wrong with my installation or something configured improperly?

Please help,

<3 sumohax0r
 


Have you talked to MT support? I'm sure they can tell you the cause of the crash. That said, 1 GB of RAM is not a lot... so that's where I'd look first; even if you're not using all of it you may have spikes that cause it to crash. Is the memory burst-able or fixed?

Stan: there is nothing wrong with CentOS... It's a well respected linux distro.

Dan
 
Agree - 1G RAM is probably too little.
Can you specify the 30K impressions - per minute or ??


Quick wins:
* install PHP opcode caches/accelerators like APC, XCache, ...
* check your mysql my.conf !!!
* switch openX media delivery to memcache! (openX v. >2.8)
 
other than that - find bottlenecks with profiling (xdebug, logfiles,...)
optimize your php stack -
* lighttpd/nginx > apache
* use reverse proxies like squid/varnish
 
You could optimize apache, php, etc, but I think while you're trying to run OpenX on the same box as other stuff (especially Plesk), you're never going to be happy, unless your ad volume is low, and stays low (peaks in your ad delivery could cause OpenX to bring down the machine and your other web sites/applications with it).

Much better: take OpenX off that box, get another box for $50/month and run OpenX on that box. You can then optimize this box specifically for OpenX, and upgrade it as your ad volume increases. Ad traffic peaks will only cause latency for your ad traffic.

m
 
true
separating your apps would be best

anyhow - installing APC or improving your stack can work wonders.
 
The best way to host an adserver is to use one of the many "cloud" hosting services out there like Mosso or Amazon. You will pay more when you are sending lots of traffic but you can scale very quickly. Then you can focus on the things that actually make you money. Think about the amount of money you lost on just that single downtime, you could have probably paid a months worth of downtime with it.

It really only makes sense to use your own servers if you have consistent traffic volumes (an optimized dedicated server costs 1/10th of typical cloud options) otherwise you will pay with either downtimes or under utilized servers.

Edit to add Mosso is now rackspacecloud.
 
danushman said:
Have you talked to MT support? I'm sure they can tell you the cause of the crash. That said, 1 GB of RAM is not a lot... so that's where I'd look first; even if you're not using all of it you may have spikes that cause it to crash. Is the memory burst-able or fixed?

Stan: there is nothing wrong with CentOS... It's a well respected linux distro.

Dan
Click to expand...

I'm fully well-aware and was poking fun.
 
Thanks for all the help everyone, I've since decided to give Adshuffle a try, and other then they're complex setup process it seems to be working ok.

I would really like to run the ad server myself so I will try some of these tips.

@dhamma - It was 30k impressions in about 10 minutes. and I will be in contact with MT to get some support on the issue.

Thanks for your help everyone, I'll definitely be looking into all of these options.
 
sumohax0r said:
Thanks for all the help everyone, I've since decided to give Adshuffle a try, and other then they're complex setup process it seems to be working ok.

I would really like to run the ad server myself so I will try some of these tips.

@dhamma - It was 30k impressions in about 10 minutes. and I will be in contact with MT to get some support on the issue.

Thanks for your help everyone, I'll definitely be looking into all of these options.
Click to expand...

What was your experience with AdShuffle?
 
Those are terribly weak specs for an ad server. 30k hits in 10 mins is about 17 hits/sec. Which is what I would guestimate the max range of what a xeon 5420 with only 1gb of memory can do. IMO switching the platform will give you minimal results if any.
 
need help setting up openx that doesn't suck? hit me up on aim.

i think these guys are right, that your problem is 1gb ram, and you need a bigger server, but regardless, i pretty much set up clustered adservers in my sleep, so you can talk to me online and we'll figure out what you need.
 
Holy crap people, are you all mad?

1GB is plenty for such tiny application and measly traffic numbers.

17hits/second is not a big deal. Just a few years ago, people had servers with half that ram and ten times those traffic numbers without any problems.

I don't know what OpenX is, but I'll make a couple of assumptions. One of the assumptions is that that software is not a complete piece of shit, and if it stores data, the schema is not completely retarded and properly indexed.

If that's the case, then you can easily handle more than that without your server thrashing.

I'm going to make another assumption, I'm guessing that you are using apache.

What you need to do is make sure apache isn't hogging all of your ram needlessly.

Let me explain some theory to you, and then you can have one these script kiddies do their "guru" shit.

First, apache by default has a keep alive timeout of 5 seconds. That means for each request (for clients that advertise keep alive, which is pretty much every modern browser), apache will maintain the connection for 5 seconds.

That means, the process (if you are using prefork) will be used for that connection for the duration of 5 seconds after the request had been served.

For ad serving you don't need keep alive at all. For ad serving + redirect tracking, keep alive would help, but only for 1-2 seconds, max. People will slower connections won't benefit, and people on broadband will be done in less than that.

Now here is why the box is thrashing with such measly traffic levels.

Let's assume the mentioned average 17 hits per second with spikes of up to 25 hits per second.

Multiply that by 5 (seconds) and you get 125 processes occupied by nothing more than waiting for follow up requests that never come.

Each apache process (that loads up php or whatever else) can take 10-20MB of ram.

That's 1.2GB-2.5GB of ram use right there.... all spent on not doing any work at all.

And once it starts swapping and more requests come in, the box is thrashing and there is no going back.

So first try disabling keepalive completely. That will immediately drop your number of processes from 125 to 20-30.

Then, check the average size of non-shared memory taken up by each apache process and use that number instead of 10-20MB I mentioned above.

Based on that number, set your max clients so that apache never actually uses up all your available ram. I would guess something along the lines of 64 processes would be good. Make sure there is enough ram for other processes like dbms and whatever else you have running.

Make sure that listen backlog is set at default (I believe 511) or higher. But really, the default setting should be fine.

Then, set set max spare servers to 30 (keep min spare at default). That will prevent needless forking/killing over and over.

In production, the worse case is you get a ton of people on dial up who would hog all your available apache processes. In reality, with 64, that's not going to happen.

On top of that, even if it gets to a point of hitting max clients, if you listen back log is sufficiently large, your visitors/eyeballs won't even notice anything aside from a slightly increased latency on the response. But that's a lot better than completely going down.

Run a stress test and see how your box behaves.

Most likely, your setup will no longer be thrashing the box, but will become cpu bound.

If that turns out to be the case, then you can start on optimizing whatever is the most cpu intensive component at this point.

Alternatively, reduce tne max number of hits for keepalive to 3 (or whatever is the number of hits needed for tracking + ad serving + redirecting) from the default of 100 and timeout to 2 seconds. That will work just as well.
 
from my experience openx with a normal 2-3 ads per page uses about the same resources as a typical wordpress install. So my advice is to purchase your server accordingly.
 
You must log in or register to reply here.
Top Bottom