I work at a company with a high volume web site (you know the drill:
LVS directors managing a farm of webservers (with php accelerators) to
a few database servers (MySQL if you want to know -- anyone happen to
have experience with Emic clustering for MySQL, btw? OK, back on
topic...)
PHP4 and Apache1 have served us well in the past. It had flaws (like
database connection pooling), but was stable and we had workarounds.
There hasn't been much in Apache 2.0 that deserved an upgrade. This is
changing.
The main item is that we need Keep-Alive connections on the web
server. It used to be something that was a nice addition -- images
downloaded a little faster, etc. But our new software uses AJAX type
stuff and now latency is a fundamental need rather than a nice
feature. There is design issue with Apache in that a single connection
is tied to a single processes (in Apache1 or Apache2 Prefork) or
thread (in Apache2 worker). (BTW: MySQL also has this deficiency).
This is changing in Apache 2.1/2.2 with the event MPM. It has a
separate thread for each thread group (process) that handles the open
Keep-Alive connection. It works very well (though it does require a
2.6 Linux kernel so we'll have lots of upgrades to stage out).
Anyone dealing with volume knows that Keep-Alive on Apache1 will kill
your setup. Adding web hardware increases the need for db hardware and
the whole setup becomes more costly and complicated that it needs to
be.
The setup we are striving for is to keep connections open all the way
down the chain to the database. Ideally this would mean that we would
use Apache 2.2 with the event MPM to hold user connections. Then the
PHP threads would ideally hold persistent connections to the databases
(either through using the Apache 2.1 mod_dbd or doing it in PHP
itself).
An aside for the moment: while the persistent connection functionality
in the mysql extension seems rather useless at first since it its per
process and not per server, there is still a use -- when it is used in
conjunction with connection pooling software running on the webserver
(which keeps its own -- far less -- actual connections open to the db
server). The new mysqli extension therefore can't really work with
connection pooling as well in this manner and is even less useful than
the current situation. Sigh.
My ideal also has MySQL breaking the one connection to one thread idea
as Apache is doing in the event MPM which would at least reduce the
problems with the lack of connection pooling with PHP (though not
completely).
Almost all of this could be fixed in PHP if it were to run well in a
threaded environment. Persistent connection pooling would be easier to
accomplish, we could have keep alive turned on all the time. With PHP
officially supporting threaded environments the various accelerators
would likely support it as well (why isn't this built in to PHP? It is
way too pokey without it).
I have read and understand that there are a number of issues:
- Problems with the Apache2 sapi
- Problems with thread safety in the Zend Engine or PHP itself
- Problems with thread-safety of modules
I don't think it is the PHP developers responsibility to make all
modules thread safe themselves (though if the Apache group wanted more
people using the worker MPM, they should have considered this option
long ago). But it could be possible to fix #1 and #2 (if they are even
issues anymore, my research on this is scattered). And there are
potential ways to deal with #3. Here are my two suggestions:
-
Make every module that is thread safe add a symbol to denote it as
such. Then when making the source we can give a warning about which
modules are not thread safe. -
Create a global extension thread mutex and put all the calls to any
extension though it. This will not fix the issue, sadly. But it will
accomplish two things: it will reduce the likelyhood of problems and
it will make the use of certain extensions slow. So between calling
out their name in #1 and making them slow in #2, there might be enough
attention on them to convince the maintainers to make them
thread-safe.
Or maybe not.
Please don't take this message in any way other than the thoughts of a
webmaster that is dealing with these issues. The desire for low
latency is so strong that we may just abondon PHP some day because of
it. Maybe someone can hack Apache1 to includes new connection handing?
At least for now, another website has lent us their custom software
for connection pooling, so even if we degrade its connections to php5
(with PDO or mysqli) we will be OK with just an Apache1 hack. Alas,
the Apache group isn't too interested in rewriting parts of Apache1.
Who can blame them? Anyone want to backport PDO to PHP4?
Well anyhow, thanks for reading and your consideration. Its amazing
how one newly popular thing (AJAX stuff in this case) can change our
whole view of the software platform landscape.
steve
steve roussey wrote:
I have read and understand that there are a number of issues:
- Problems with the Apache2 sapi
- Problems with thread safety in the Zend Engine or PHP itself
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.
My stuff is far bigger and busier than yours, and I have absolutely no
intention to go threaded anytime soon.
-Rasmus
Nonetheless, Rasmus, isn't it something to work on? Like Steve said, #1 and
#2 can be accomplished, and that would at least be a good start. You are
right about the libraries, but in time, even these may become thread safe
(of course, likely not by anyone from the PHP-community). Why is this
thread-safety such a taboo for many of the people here? Don't get me wrong,
I know it can't be done in one go, but a start could be made, right? Perhaps
that start could be made from PHP6 on? It doesn't have to be officially
thread safe, because of all the libraries, but it could be thread-safer...
Ron
"Rasmus Lerdorf" rasmus@lerdorf.com wrote in message
news:42F82AE3.6070602@lerdorf.com...
steve roussey wrote:
I have read and understand that there are a number of issues:
- Problems with the Apache2 sapi
- Problems with thread safety in the Zend Engine or PHP itself
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.My stuff is far bigger and busier than yours, and I have absolutely no
intention to go threaded anytime soon.-Rasmus
If we find thread safety problems in PHP we will fix them, of course.
It's not like we ignore them. Same goes for extensions, except for
those which are specifically designed to access libraries that show no
signs of ever becoming thread safe.
At the same time, you need to recognize that developers work on what
interests them and what they personally have a need for. I can't speak
for everyone, but a good number of the core developers lean towards the
robustness of a multi-process architecture. Having separate memory
spaces makes everything easier to debug and when something does go wrong
it doesn't take down the whole server. These are both very compelling
reasons to stick with this architecture.
-Rasmus
Ron Korving wrote:
Nonetheless, Rasmus, isn't it something to work on? Like Steve said, #1 and
#2 can be accomplished, and that would at least be a good start. You are
right about the libraries, but in time, even these may become thread safe
(of course, likely not by anyone from the PHP-community). Why is this
thread-safety such a taboo for many of the people here? Don't get me wrong,
I know it can't be done in one go, but a start could be made, right? Perhaps
that start could be made from PHP6 on? It doesn't have to be officially
thread safe, because of all the libraries, but it could be thread-safer...Ron
"Rasmus Lerdorf" rasmus@lerdorf.com wrote in message
news:42F82AE3.6070602@lerdorf.com...steve roussey wrote:
I have read and understand that there are a number of issues:
- Problems with the Apache2 sapi
- Problems with thread safety in the Zend Engine or PHP itself
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.My stuff is far bigger and busier than yours, and I have absolutely no
intention to go threaded anytime soon.-Rasmus
Hi Ron,
Nonetheless, Rasmus, isn't it something to work on? Like Steve said, #1 and
#2 can be accomplished, and that would at least be a good start. You are
right about the libraries, but in time, even these may become thread safe
(of course, likely not by anyone from the PHP-community). Why is this
thread-safety such a taboo for many of the people here? Don't get me wrong,
I know it can't be done in one go, but a start could be made, right?
Perhaps that start could be made from PHP6 on? It doesn't have to be
officially thread safe, because of all the libraries, but it could be
thread-safer...
PHP itself is thread-safe. The problem is in the libraries. Rasmus once wrote
a nice mail to this list about these problems:
http://news.php.net/php.internals/10491
johannes
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.
OK, I am confused. I always thought (an assumption with nothing to
back it up, now that I think about it) that the core part of PHP was
thread safe and that it was the third party libraries included from
extensions that were at issue.
So if I used mysqli then it was at the mercy of the mysql library. So
if the mysqli maintainer knew that that library was safe and their
extension was safe then they could mark it as safe. All others would
go through a single mutex, etc...
But if PHP core uses a library and has no idea if it is safe or not
(like glibc, though would run under such a system?) then I get your
point. I hadn't considered it before. I suppose that is why MySQL
statically links in such things that it knows works, not leaving
anything to chance.
I guess PHP's greatest strength is also its weakness, PHP can never
realistically be thread safe and we'll all just deal with it.
Really, it has never been an issue before. I'm not a flag waver for
threads! I've been perfectly happy with PHP4 and Apache1. It is just
the Keep-Alive and connection pooling issues would more easily work
themselves out in a threaded environment. Its a clean solution (from
the point of view of using PHP, developing that is a different issue).
I suppose adding in layers to fix these issues is the best direction
to take, like separate connection pooling software to databases and
another separate application to the same in reverse with keep-alive (a
proxy would work here). With both "bookends" running on the same
server and using domain sockets to PHP, it should work fine and is no
longer a subject for this list. :)
My stuff is far bigger and busier than yours
I know my name can be easily traced to a website I own that only gets
about 10M pv/day but I also work as a consultant for another that gets
an order of magnitude more. So perhaps you do get more then them and
get 1B pv/day and your dad can beat up my dad. Ug. Strike that. I'm
just frustrated and stressed. (My own website doesn't even need the db
connection pooling just yet. Though it could use Keep-Alive..)
I'll leave this discussion only with this:
I love PHP and appreciate what everyone has contributed over the
years. Thanks! (And now I'll shut up.)
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.OK, I am confused. I always thought (an assumption with nothing to
back it up, now that I think about it) that the core part of PHP was
thread safe and that it was the third party libraries included from
extensions that were at issue.So if I used mysqli then it was at the mercy of the mysql library. So
if the mysqli maintainer knew that that library was safe and their
extension was safe then they could mark it as safe. All others would
go through a single mutex, etc...But if PHP core uses a library and has no idea if it is safe or not
(like glibc, though would run under such a system?) then I get your
point. I hadn't considered it before. I suppose that is why MySQL
statically links in such things that it knows works, not leaving
anything to chance.
This can affect you even if you're not statically linked.
Dynamically loaded libraries run in your address space and have the
same ability to mess you up as statically linked libraries.
Otherwise your understanding is spot-on.
Here there be dragons.
George
My stuff is far bigger and busier than yours
My spewing of stuff...
I often write things to make me feel better and delete them before
hitting send. Sort of stream of consciousness that I edit before
sending. IRasmus, apologize for being an ass.
-steve
steve roussey wrote:
I know my name can be easily traced to a website I own that only gets
about 10M pv/day but I also work as a consultant for another that gets
an order of magnitude more. So perhaps you do get more then them and
get 1B pv/day and your dad can beat up my dad. Ug. Strike that. I'm
just frustrated and stressed. (My own website doesn't even need the db
connection pooling just yet. Though it could use Keep-Alive..)
Well, I can generally safely say that unless you also happen to work at
Yahoo! in which case we are tied. ;)
-Rasmus
We've made a start a long time ago, and we do our best to keep all of PHP
thread-safe. On paper it is. When a bug appears we fix it. Is it a risk to
run it... I think it is but I have seen people running it successfully on
IIS for a long time. (I still think that's a bad idea though).
Andi
At 07:09 PM 8/9/2005 +0200, Ron Korving wrote:
Nonetheless, Rasmus, isn't it something to work on? Like Steve said, #1 and
#2 can be accomplished, and that would at least be a good start. You are
right about the libraries, but in time, even these may become thread safe
(of course, likely not by anyone from the PHP-community). Why is this
thread-safety such a taboo for many of the people here? Don't get me wrong,
I know it can't be done in one go, but a start could be made, right? Perhaps
that start could be made from PHP6 on? It doesn't have to be officially
thread safe, because of all the libraries, but it could be thread-safer...Ron
"Rasmus Lerdorf" rasmus@lerdorf.com wrote in message
news:42F82AE3.6070602@lerdorf.com...steve roussey wrote:
I have read and understand that there are a number of issues:
- Problems with the Apache2 sapi
- Problems with thread safety in the Zend Engine or PHP itself
- Problems with thread-safety of modules
You missed the most serious one. Thread safety problems in random
libraries you link in and we have absolutely no control over those.My stuff is far bigger and busier than yours, and I have absolutely no
intention to go threaded anytime soon.-Rasmus
steve roussey wrote:
The setup we are striving for is to keep connections open all the way
down the chain to the database. Ideally this would mean that we would
use Apache 2.2 with the event MPM to hold user connections. Then the
PHP threads would ideally hold persistent connections to the databases
(either through using the Apache 2.1 mod_dbd or doing it in PHP
itself).
Have you considered using lighttpd with fastcgi loadbalancing and
caching using mod_cml (cache meta language)?
We have a similar problem here, and that's how we're going to solve it:
By using lighttpd with fastcgi we seperate the webserver process from
php processes (which could even work on other machines) which saves a
lot of resources (memory/cpu/load) and enhance response-time for static
files.
For simple dynamic requests we can use mod_cml, which can "communicate"
with PHP using memcached or the filesystem. Only if mod_cml cannot
response to the request using cached data, php is needed. That speeds
things up a lot, since cache-hits in mod_cml are not much slower than
static files (because php is not touched in any way).
It works perfectly with keep-alive.
The connections from lighttpd process to fastcgi php processes are also
persistent. It's no problem to use a php opcode cache. Perhaps
db-connection pooling isn't an issue anymore, because the number of
php-processes (= number of db connections) can be reduced drastically.
Perhaps you can use apc_store()... or memcached to cache results of DB
queries.
http://trac.lighttpd.net/trac/wiki/CacheMetaLanguage
http://trac.lighttpd.net/trac/wiki/TutorialLighttpdAndPHP
http://trac.lighttpd.net/trac/wiki/MigratingFromApache
However, I don't think mod_php on a threaded webserver is a good idea,
least of all for a setup under high load because of the arguments
already mentioned in this thread and because most developers don't use
threaded setups today.
best regards,
Andreas
By using lighttpd with fastcgi we seperate the webserver process from
php processes (which could even work on other machines)...
Someone else emailed me about using FastCGI with Apache 2.1/event but
I just figured that there would be a significant slowdown using
FastCGI rather than a module/handler. (Currently I compile PHP into
Apache statically and turn off Apache's dynamic module loading ability
-- something I couldn't figure out in Apache2). What is your
experience with FastCGI?
For simple dynamic requests we can use mod_cml
Sadly, this won't work with any system I manage since we do a lot of
stat reporting, even on a per user basis. And we need to have PHP
loaded to know the user, etc...
Still, I looked at lighttpd and it looks promising. The one thing that
started all of this was Apache 2.1's event MPM that used a single
thread to handle all open Keep-Alives looked very efficient.
steve roussey wrote:
By using lighttpd with fastcgi we seperate the webserver process from
php processes (which could even work on other machines)...Someone else emailed me about using FastCGI with Apache 2.1/event but
I just figured that there would be a significant slowdown using
FastCGI rather than a module/handler. (Currently I compile PHP into
Apache statically and turn off Apache's dynamic module loading ability
-- something I couldn't figure out in Apache2). What is your
experience with FastCGI?
PHP by default compiles as a non-pic shared library now which is just as
fast as a static build inside Apache since it is the pic stuff that
slows down a DSO. So there is really no need for static builds anymore,
unless you happen to be on a fringe OS that doesn't support non-pic
shared libs.
Still, I looked at lighttpd and it looks promising. The one thing that
started all of this was Apache 2.1's event MPM that used a single
thread to handle all open Keep-Alives looked very efficient.
I think you are probably better off solving this in a lightweight
frontend process. Chances are you are going to need lingerd if you go
keepalive, so perhaps the real solution is to make lingerd handle not
just the shutdown, but also the startup of the request.
-Rasmus
PHP by default compiles as a non-pic shared library now which is just as
fast as a static build inside Apache since it is the pic stuff that
slows down a DSO. So there is really no need for static builds anymore,
unless you happen to be on a fringe OS that doesn't support non-pic
shared libs.
This is good to know. I guess it is time to rewrite my
build-a-new-webserver script. It has seen changes over the years but
not a comprehensive reevaluation.
Still, I looked at lighttpd and it looks promising. The one thing that
started all of this was Apache 2.1's event MPM that used a single
thread to handle all open Keep-Alives looked very efficient.I think you are probably better off solving this in a lightweight
frontend process. Chances are you are going to need lingerd if you go
keepalive, so perhaps the real solution is to make lingerd handle not
just the shutdown, but also the startup of the request.
You know, I remember considering lingerd a long time ago... and I feel
like an idiot for not using all these years! If it is not in my script
it doesn't cross my mind. So I have that on today's todo list. (This
seems like something Apache2 should do automatically in its threaded
MPMs, not that we would be using mod_php here or anything, but maybe
I am confused by your statement above, so I have tried not to email
back until I could find more information, but I could not. In the
lingerd website it says "lingerd can only do an effective job if HTTP
Keep-Alives are turned off" which is confusing when compared to your
statement above. Unless you are combining it with the lightweight
process (I assume a proxy server). Then it makes sense. Except for the
part about having lingerd hande the startup of the request, at which
point I'm clueless again.
steve roussey wrote:
PHP by default compiles as a non-pic shared library now which is just as
fast as a static build inside Apache since it is the pic stuff that
slows down a DSO. So there is really no need for static builds anymore,
unless you happen to be on a fringe OS that doesn't support non-pic
shared libs.This is good to know. I guess it is time to rewrite my
build-a-new-webserver script. It has seen changes over the years but
not a comprehensive reevaluation.Still, I looked at lighttpd and it looks promising. The one thing that
started all of this was Apache 2.1's event MPM that used a single
thread to handle all open Keep-Alives looked very efficient.I think you are probably better off solving this in a lightweight
frontend process. Chances are you are going to need lingerd if you go
keepalive, so perhaps the real solution is to make lingerd handle not
just the shutdown, but also the startup of the request.You know, I remember considering lingerd a long time ago... and I feel
like an idiot for not using all these years! If it is not in my script
it doesn't cross my mind. So I have that on today's todo list. (This
seems like something Apache2 should do automatically in its threaded
MPMs, not that we would be using mod_php here or anything, but maybeI am confused by your statement above, so I have tried not to email
back until I could find more information, but I could not. In the
lingerd website it says "lingerd can only do an effective job if HTTP
Keep-Alives are turned off" which is confusing when compared to your
statement above. Unless you are combining it with the lightweight
process (I assume a proxy server). Then it makes sense. Except for the
part about having lingerd hande the startup of the request, at which
point I'm clueless again.
Well, I don't know about the specific lingerd implementation, but the
need for SO_LINGER sockets become more critical when you turn on
keepalive because the chances of closing a socket when the client is
sending or about to send something becomes much higher the longer you
sit around with the socket open in a keepalive situation. And if you
eventually move to support pipelined requests (if browsers would ever
support that) then you absolutely must use SO_LINGER sockets. SO_LINGER
sockets mean that we close down just one end of the socket and linger
around waiting for the client to flush any data (which is then thrown
out) and then finally shut down everything.
The big problem is that most operating systems, for reasons I have never
understood, block on the close when SO_LINGER is set on it. I always
thought that made absolutely no sense and is something the damn kernel's
stack should just take care of, but it doesn't, and you end up sitting
around twiddling your thumbs in userspace waiting for the lingering
socket to shut down. Apache has a workaround called lingering_close()
that tries to address broken SO_LINGER implementations, but it also blocks.
So, lingerd is supposed to help this situation by taking over the socket
closing. Apache hands off to lingerd and can move onto the next request
and lingerd sits around waiting for the sockets to close down. I don't
really see a technical reason why lingerd cares whether the connection
was a keepalive connection or not. Seems to me like a simple
implementation detail of hooking the socket handoff into the right place
in Apache. If you turn the concept of lingerd upside down and bounce
socket descriptors back and forth between it and Apache between
keepalive requests you can free up Apache from any sort of sitting
around waiting on socket timeouts or linger outs. The last thing you
want is a heavy single-threaded process like Apache sitting around
waiting on some socket event. It should be crunching out pages as fast
as possible not waiting on a keepalive or SO_LINGER timeout.
-Rasmus
Just a couple last notes on lingering:
o Apache 2+ uses SO_LINGER by default if it defined for that system.
Apache 1 will only use it if you define USE_SO_LINGER (I suppose in
configure). Apache2 has all sorts of stuff in the comments of the code
and in the manual which is just wrong. Its all from Apache1 and does
not reflect on Apache2's implementation. I wish they just erased it
instead.
o As far as I can tell, current Linux will block until all data is
sent, then return (doing the actual closing part in the background)
with SO_LINGER.
o Lingerd caused my apache setup to crash. It was worth a try if it
didn't take much effort, but not worth fixing.
If you turn the concept of lingerd upside down and bounce
socket descriptors back and forth between it and Apache between
keepalive requests you can free up Apache from any sort of sitting
around waiting on socket timeouts or linger outs. The last thing you
want is a heavy single-threaded process like Apache sitting around
waiting on some socket event. It should be crunching out pages as fast
as possible not waiting on a keepalive orSO_LINGERtimeout.
This actually sounds like an argument for NOT using mod_php. It sounds
like an argument for using Apache2 or lighttpd or xyz in conjection
with FastCGI. (Or a proxy arangement, which I've done, though in my
personal case, I like to get the same scaling with less machines since
I have to buy the machines). In that case, the webserver can be made
lightweight (not sure how lightweight Apache2 can be, but who knows?)
handling many open connections. Then there is a (much) smaller number
of heavy PHP processes in FastCGI.
I'm going to put a test together next week and have Apache1/mod_php,
Apache2/worker (not event)/FastCGI, and lighttpd/FastCGI run head to
head and see what happens. Maybe I'll even try with KeepAlive too
(though not with Apache1/mod_php since I already know that won't
work). I'll post my results back here if anyone is interested.
Andreas -- can you email me off list? I wouldn't mind some help
setting up lighttpd/FastCGI. Thanks!
-steve
steve roussey wrote:
This actually sounds like an argument for NOT using mod_php. It sounds
like an argument for using Apache2 or lighttpd or xyz in conjection
with FastCGI. (Or a proxy arangement, which I've done, though in my
personal case, I like to get the same scaling with less machines since
I have to buy the machines). In that case, the webserver can be made
lightweight (not sure how lightweight Apache2 can be, but who knows?)
handling many open connections. Then there is a (much) smaller number
of heavy PHP processes in FastCGI.
Well, I am not sure about your conclusion there. Generally you don't
want trivial requests going through the heavyweight Apache process, so
you can either try to make Apache less heavyweight and separate out the
dynamic stuff which is what you are suggesting, or you can separate out
the trivial requests. The large players do the latter by Akamizing all
their static content or the trivially dynamic stuff and only handle
heavy requests on their own servers. For smaller players the common
solution is to have a separate set of servers doing static requests.
thttpd, Tux, lighttpd, etc. which are easier to strip down than the
heavier (and more flexible) Apache server.
-Rasmus
Yes, you are quite correct in that a very large site (Yahoo, Google,
etc) will use a caching ISP (aka Akami). In fact, I imagine that it
would be a completely separate domain name so there would be no
cookies and everyone down the chain can easily cache the content as
well. Doesn't work for all object content (great for images) since JS
for example might loose its ability to set cookies.
I'm not really considering these things anyhow, since 99% of what I
work with is access controlled or could be at a moments notice. It all
needs to be logged as well, down to the things like images.
So I'm really just looking at dynamic requests. It bears noting that
usually a set of servers serving static content could use KeepAlive
and the dynamic ones not, and a lot of this discussion is meaningless.
But, if you are say, Google Maps, and doing "AJAX" type stuff (or
chat, if only more browsers supported multipart responses from your
XMLHttpRequest object) then the situation changes -- you need the
Keep-Alive connections for all your dynamic connections too. And now
I'm back at the beginning of the thread.
I have my doubts about stripping Apache2 down to something
'lightweight'. I have my own thoughts on how that project was
engineered. But there is nothing like a SmackDown to see where the
cookies crumble.
Thanks!
-steve--
Well, I am not sure about your conclusion there. Generally you don't
want trivial requests going through the heavyweight Apache process, so
you can either try to make Apache less heavyweight and separate out the
dynamic stuff which is what you are suggesting, or you can separate out
the trivial requests. The large players do the latter by Akamizing all
their static content or the trivially dynamic stuff and only handle
heavy requests on their own servers. For smaller players the common
solution is to have a separate set of servers doing static requests.
thttpd, Tux, lighttpd, etc. which are easier to strip down than the
heavier (and more flexible) Apache server.-Rasmus
Just a couple last notes on lingering:
o Apache 2+ uses
SO_LINGERby default if it defined for that system.
Apache 1 will only use it if you define USE_SO_LINGER (I suppose in
configure). Apache2 has all sorts of stuff in the comments of the code
and in the manual which is just wrong. Its all from Apache1 and does
not reflect on Apache2's implementation. I wish they just erased it
instead.
sorry, but i don't see any SO_LINGER, except some comments in apache2
source code, even you want to enable it by configure option. can you
bring me to the source or document where you're told that apache2 uses
SO_LINGER?o As far as I can tell, current Linux will block until all data is
sent, then return (doing the actual closing part in the background)
with SO_LINGER.o Lingerd caused my apache setup to crash. It was worth a try if it
didn't take much effort, but not worth fixing.
srclib/apr/network_io/unix/sockopt.c: if
(setsockopt(sock->socketdes, SOL_SOCKET, SO_LINGER, (char *) &li,
sizeof(struct linger)) == -1)
...
It gets set if APR_SO_LINGER, is set which it is:
srclib/apr/include/apr_network_io.h:#define APR_SO_LINGER 1
/**< Linger */
and if SO_LINGER is set, which it is inside the linux socket include
file (the included bits one, but I'm going by memory here).
sorry, but i don't see any SO_LINGER, except some comments in apache2
source code, even you want to enable it by configure option. can you
bring me to the source or document where you're told that apache2 uses
SO_LINGER?
o Apache 2+ uses
SO_LINGERby default if it defined for that system.
Really? We just did a around of discussion/debugging on this at work
and I found that it uses ap_lingering_close which is like the
lingering_close function in 1.3.
Apache 1 will only use it if you define USE_SO_LINGER (I suppose in
configure).
Right and the default is to use lingering_close, unless NO_LINGCLOSE is
defined on some os-es like SUNOS4, IRIX, NEXT, AUX3, UW.
Apache2 has all sorts of stuff in the comments of the code
and in the manual which is just wrong. Its all from Apache1 and does
not reflect on Apache2's implementation. I wish they just erased it
instead.
I just wish people would fix the TCP stack, it is broken damn it.
o As far as I can tell, current Linux will block until all data is
sent, then return (doing the actual closing part in the background)
with SO_LINGER.
This is what we found for Linux and FreeBSD, kernel guys say it is not
a bug in the TCP stack sense everybody blocks.
o Lingerd caused my apache setup to crash. It was worth a try if it
didn't take much effort, but not worth fixing.
Are people really finding that they need lingerd?
For 1.3 we have keepalive off and build with NO_LINGCLOSE, which means
be blast the data to the kernel buffer (set large) close the socket and
move on. We have been doing this for at least 3+ years and I have
never heard any complaints and we server a lot of requests/client (I am
tied with Rasmus).
Brian
Really? We just did a around of discussion/debugging on this at work
and I found that it uses ap_lingering_close which is like the
lingering_close function in 1.3.
:)
Yes, it does do ap_lingering_close, and it sets SO_LINGER. I have no
idea why they do both, I only assume they know what they are doing and
it works out for the best. I have my ideas, but they are just
postulates... I'm surprised it is an ap call and not an apr call, but
I am not a web server programmer!
o Lingerd caused my apache setup to crash. It was worth a try if it
didn't take much effort, but not worth fixing.Are people really finding that they need lingerd?
For 1.3 we have keepalive off and build with NO_LINGCLOSE, which means
be blast the data to the kernel buffer (set large) close the socket and
move on. We have been doing this for at least 3+ years and I have
never heard any complaints and we server a lot of requests/client (I am
tied with Rasmus).
I should give that a test. But the value to me would be temporary,
since I want to move to Keep-Alive under the same hardware layout.
-steve
Really? We just did a around of discussion/debugging on this at work
and I found that it uses ap_lingering_close which is like the
lingering_close function in 1.3.:)
Yes, it does do ap_lingering_close, and it sets SO_LINGER. I have no
idea why they do both, I only assume they know what they are doing and
it works out for the best.
I see the code for SO_LINGER in apr_socket_opt_set, but I don't see any
place in the server code that calls apr_socket_opt_set with
APR_SO_LINGER.
Running gdb I see APR_SO_KEEPALIVE, APR_SO_REUSEADDR and
APR_TCP_NODELAY passed to apr_socket_opt_set but not APR_SO_LINGER.
(This is based on the svn of httpd with no changes)
I'm surprised it is an ap call and not an apr call, but
I am not a web server programmer!
I agree, it took me a while to find the call again because I keep
searching for apr_.
Brian
I see the code for
SO_LINGERin apr_socket_opt_set, but I don't see any
place in the server code that calls apr_socket_opt_set with
APR_SO_LINGER.
I stand corrected. I should have used gdb and tested that before I
posted. So they put code in there to set it, but they don't have way
to reach it. A "grep APR_SO_LINGER -rI httpd-2.1.6-alpha" shows that
it is not mentioned much and looking through again this morning when
fresh, it never as a change to get to apr_socket_opt_set. So, I guess,
Apache2 has even less support for SO_LINGER than Apache1. Can't even
set a define in configure, must hack the code. Anyhow, sorry about the
incorect information.
I learned a lot through this thread and would like to thank everyone
for their thoughts, information, and ideas. I'll return next week with
some stats on FastCGI (various) vs mod_php.
-steve
steve roussey wrote:
Someone else emailed me about using FastCGI with Apache 2.1/event but
I just figured that there would be a significant slowdown using
FastCGI rather than a module/handler. (Currently I compile PHP into
Apache statically and turn off Apache's dynamic module loading ability
-- something I couldn't figure out in Apache2). What is your
experience with FastCGI?
There are some different benchmarks with lighttpd+php:
http://trac.lighttpd.net/trac/wiki/
But I only trust my own benchmarks, and here I couldn't measure a really
big difference, lighttpd + fastcgi seems to be slightly faster than
Apache 1.3 + mod_php with my setup. But I think it depends on your
application and setup. A big difference I noticed was used cpu, memory
and load - lighttpd was by far more efficient here.
If you're benchmarking lighttpd you should have a look at the following
chapter from documentation:
http://lighttpd.net/documentation/performance.html
Still, I looked at lighttpd and it looks promising. The one thing that
started all of this was Apache 2.1's event MPM that used a single
thread to handle all open Keep-Alives looked very efficient.
lighttpd does the same, it's a non-blocking webserver with only one
process. The one process cares about keep-alives... and delegates
handling of PHP-Requests to a number of loadbalanced, persistent PHP
Processes using FastCGI.
Andreas