# adding low level file handling stuff - Thread: https://externals.io/message/52091 - Original archive: https://news-web.php.net/php.internals/52091 - Started by Ferenc Kovacs on 2011-04-30T12:04:12+00:00 - Messages: 13 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #52091 — Ferenc Kovacs — 2011-04-30T12:04:12+00:00 URL: https://externals.io/message/52091#52091 Hi. recently I found a nice blogpost about how to properly daemonize a php daemon: http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ I've noticed in this article, that you can replace/redirect the STDIN/STDOUT/STDERR from inside of your script, you have to close them, and open in the correct order. It works (at least on linux), because the opened files will have the 1,2,3 FDs, because the OS assign the lowest available FD, which will happen to be the required ones. I would love the idea to replace the STDIN/STDOUT/STDERR (and maybe any open streams/fps) from my script, but in this form I see many possible pitfalls. - if the STDIN/STDOUT/STDERR isn't the lowest available FDs, then you get wrong FD-s, and you have no sane way to tell it(you can't get the FD for a fp/stream from php). - if the OS doesn't follows the Unix/Posix spec: "The *open*() function shall return a file descriptor for the named file that is the lowest file descriptor not currently open for that process." then you get wrong FD-s, and you have no sane way to tell it. - if something opens an FD between the fclose and fopen call(callback for register_tick_function and pcntl_signal can happen between those calls), then you get wrong FD-s, and you have no sane way to tell it. - if you close a fp which has higher FD than the lowest available, then you get wrong FD-s, and you have no sane way to tell it. So to correctly address this, one would need a way to get the FD for a given fp and to be able to explicitly set the FD for an fp. The required C methods for this would be the fileno and dup2. I've started to look around, that maybe there are existing functions/extensions to make this work. I've found that with 5.3.6 one can open arbitrary FDs thanks to Gustavo: http://bugs.php.net/bug.php?id=53465 this is the equivalent of fdopen, which was requested by Dennis Hotson http://marc.info/?t=126854891300001&r=1&w=2 and he did implement that https://github.com/dhotson/fdopen-php (+1 for adding fdopen also, the stream based solution isn't obvious and easy to spot) But this doesn't help this problem. I checked and found that there is a pecl module, which provides some low level file handling: http://pecl.php.net/package/dio it doesn't provide fileno, and dup2, and doesn't support file pointers/streams, but I started wondering, why was this removed from the core, and moved to pecl. so I think that those methods needs to be added, but I'm not sure where would be appropriate to have methods like dup, dup2, finfo, fcntl (the dio_fcntl is far from complete). it could be added to the posix methods, or to the dio pecl extension, or a third extension, so I'm a little bit confused. I would like to know, that why do these methods lacks (just nobody needed them, or maybe it was a decision to keep out such low level methods from the core)? and if someone would make those, what should be the best place to add. btw: I've did POC implementation of fileno: https://gist.github.com/947595, but I don't have enough experience in C and php internals, so while I would like to help move things forward, but would need assistance with it. What do you think? Tyrael --- ## #52092 — Alain Williams — 2011-04-30T12:18:25+00:00 URL: https://externals.io/message/52091#52092 · In reply to: #52091 > Hi. > > recently I found a nice blogpost about how to properly daemonize a php > daemon: > http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ > I've noticed in this article, that you can replace/redirect the > STDIN/STDOUT/STDERR from inside of your script, you have to close them, and > open in the correct order. > It works (at least on linux), because the opened files will have the 1,2,3 Surely you mean: 0, 1, 2 -- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 http://www.phcomp.co.uk/ Parliament Hill Computers Ltd. Registration Information: http://www.phcomp.co.uk/contact.php #include > > […] > http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ > > […] > and > > […] > 1,2,3 > > Surely you mean: 0, 1, 2 > > sorry, right, thanks for correcting. Tyrael --- ## #52094 — Reindl Harald — 2011-04-30T12:54:17+00:00 URL: https://externals.io/message/52091#52094 · In reply to: #52091 Am 30.04.2011 14:04, schrieb Ferenc Kovacs: > Hi. > > recently I found a nice blogpost about how to properly daemonize a php > daemon: > http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ > I've noticed in this article, that you can replace/redirect the > STDIN/STDOUT/STDERR from inside of your script, you have to close them, and > open in the correct order. > It works (at least on linux), because the opened files will have the 1,2,3 > FDs, because the OS assign the lowest available FD, which will happen to be > the required ones the cleaner way is to do this in a /etc/init.d/service-script we are running a monitoring-service written in php wilh a while(1==1) as linux-daemon since years and even config-changes are no problem with an internal counter for reloading them every 10 while-runs sleep(1) is a good idea fpr each run to make sure the daemon eats not too much ressources useless --- ## #52095 — Gustavo Lopes — 2011-04-30T13:26:23+00:00 URL: https://externals.io/message/52091#52095 · In reply to: #52091 > recently I found a nice blogpost about how to properly daemonize a php > daemon: > http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ > I've noticed in this article, that you can replace/redirect the > STDIN/STDOUT/STDERR from inside of your script, you have to close them, > and open in the correct order. > It works (at least on linux), because the opened files will have the > 1,2,3 FDs, because the OS assign the lowest available FD, which will > happen to be the required ones. > > [...] > So to correctly address this, one would need a way to get the FD for a > given fp and to be able to explicitly set the FD for an fp. > The required C methods for this would be the fileno and dup2. > > I've started to look around, that maybe there are existing > functions/extensions to make this work. > I've found that with 5.3.6 one can open arbitrary FDs thanks to Gustavo: > http://bugs.php.net/bug.php?id=53465 > this is the equivalent of fdopen, which was requested by Dennis Hotson > http://marc.info/?t=126854891300001&r=1&w=2 and he did implement that > https://github.com/dhotson/fdopen-php (+1 for adding fdopen also, the > stream based solution isn't obvious and easy to spot) > > But this doesn't help this problem. > > [...] > I would like to know, that why do these methods lacks (just nobody needed > them, or maybe it was a decision to keep out such low level methods from > the core)? > and if someone would make those, what should be the best place to add. > > [...] I actually did consider adding support for an extended form of php://fd where one would specify the desired file descriptor: php://fd//. It would call dup2 instead of dup. However, I got into some trouble on shutdown because this could cause stdout to be closed ahead of time and then the output subsystem would cause either a segfault or a memory leak (can't recall). I didn't spend more than 20 minutes on this as it was not the problem I was trying to solve, so there's probably an easy solution. If you want to work on this, extending php://fd would likely be a good place. -- Gustavo Lopes --- ## #52096 — Ferenc Kovacs — 2011-04-30T15:08:19+00:00 URL: https://externals.io/message/52091#52096 · In reply to: #52094 On Sat, Apr 30, 2011 at 2:54 PM, Reindl Harald wrote: > > Am 30.04.2011 14:04, schrieb Ferenc Kovacs: > > […] > http://andytson.com/blog/2010/05/daemonising-a-php-cli-script-on-a-posix-system/ > > […] > and > > […] > 1,2,3 > > […] > be > > […] > > the cleaner way is to do this in a /etc/init.d/service-script > > in what way would a service script help this particular issue? or you mean like a wrapper script which sets the `STDIN` STDOUT STERR before launching the script? eg: nohup command < /dev/null > /dev/null 2>&1 & I find the command form much cleaner, but with the currently available tools one cannot use this, because there is no way to properly daemonize* (you can fork and posix_setsid but you cannot reopen the stdin, stdout and stderr) at least without doing a proc_open or exec * http://www.itp.uzh.ch/~dpotter/howto/daemonize > we are running a monitoring-service written in php wilh a while(1==1) > as linux-daemon since years and even config-changes are no problem with > an internal counter for reloading them every 10 while-runs > > sleep(1) is a good idea fpr each run to make sure the daemon eats not too > much ressources useless > > I agree (except while(true) is shorter, or even for(;;) but thats a little bit confusing at first :P), but this is irrevelant for this discussion, I didn't asked help on how to write a daemon app. Tyrael --- ## #52098 — Ferenc Kovacs — 2011-04-30T15:23:48+00:00 URL: https://externals.io/message/52091#52098 · In reply to: #52095 > > I actually did consider adding support for an extended form of php://fd > where one would specify the desired file descriptor: php://fd// fd>. It would call dup2 instead of dup. > > However, I got into some trouble on shutdown because this could cause > stdout to be closed ahead of time and then the output subsystem would cause > either a segfault or a memory leak (can't recall). I didn't spend more than > 20 minutes on this as it was not the problem I was trying to solve, so > there's probably an easy solution. If you want to work on this, extending > php://fd would likely be a good place. > > -- > Gustavo Lopes > > -- > > > > thanks Gustavo. as you check my mail, I also mentioned using dup2 instead of dup, for obvious reasons. adding dup2 support for the php://fd stream would be better than the current situation, but I can't see why are we trying to force everything into that. I mean everything on the http://www.php.net/manual/en/wrappers.php can be achived through a function also. there are some case when you can save a couple of lines of code with the wrappers and with http://www.php.net/manual/en/context.http.php but there is no method for opening FDs or the dup2 that you mentioned. I think that it would be more consistent with the rest of the language, and one would more likely to find fdopen than fopen('php://fd/1'); in the documentation. closing the stdout without reopening also caused problems for me in case of an error happens, but this shouldn't happen if you properly reopen it, so I would be interested what exactly happened there. Tyrael --- ## #52131 — Ferenc Kovacs — 2011-05-05T00:30:41+00:00 URL: https://externals.io/message/52091#52131 · In reply to: #52098 > I actually did consider adding support for an extended form of php://fd > > […] > thanks Gustavo. > as you check my mail, I also mentioned using dup2 instead of dup, for > obvious reasons. > adding dup2 support for the php://fd stream would be better than the > current situation, but I can't see why are we trying to force everything > into that. > I mean everything on the http://www.php.net/manual/en/wrappers.php can be > achived through a function also. > there are some case when you can save a couple of lines of code with the > wrappers and with http://www.php.net/manual/en/context.http.php but there > is no method for opening FDs or the dup2 that you mentioned. > I think that it would be more consistent with the rest of the language, and > one would more likely to find fdopen than fopen('php://fd/1'); in the > documentation. > > closing the stdout without reopening also caused problems for me in case of > an error happens, but this shouldn't happen if you properly reopen it, so I > would be interested what exactly happened there. > > Tyrael > I put together the fileno, fdopen, dup2, fclose stuff as a pecl extension(called fildes), the sourcecode available here: https://github.com/Tyrael/php-fildes with this, you can do something like this: > > > > […] > > I put together the fileno, fdopen, dup2, fclose stuff as a pecl > extension(called fildes), the sourcecode available here: > https://github.com/Tyrael/php-fildes > with this, you can do something like this: > $stdout_fd_orig = fildes_fileno(STDOUT); > fclose(STDOUT); > $stdout = fopen("out.log", "a"); > $fd_tmp = fildes_fileno($stdout); > $stdout_fd = fildes_dup2($fd_tmp, $stdout_fd_orig); > if($stdout_fd != $fd_tmp){ >         fildes_close($fd_tmp); > } > echo "Hello world!"; // this goes to out.log > I'm really a novice in C, so any feedback welcome. > particularly > here https://github.com/Tyrael/php-fildes/blob/master/fildes.c#L211 > Tyrael as somebody pointed out, the example could be simplified as: > > […] > > as somebody pointed out, the example could be simplified as: > $stdout = fopen("out.log", "a"); > $tmp_fd = fildes_fileno($stdout); > if($tmp_fd, fildes_fileno(STDOUT))<0){ >    fwrite(STDERR, "dup2 failed"); >    exit; > } > fildes_close($tmp_fd); > echo "Hello world!"; // this goes to out.log > > Could somebody review the source? as I mentioned in my previous email > I had some problem here: > https://github.com/Tyrael/php-fildes/blob/master/fildes.c#L211 > > if I define the second argument as int, the first argument will be always 0. > I have no idea whats going there. > > Tyrael > and I forgot to ask the most important thing: what do you think about the feature itself? I would really like to have it in the core, but I think it would be better to have this as functions, than adding more stream magic. I'm not sure where would this functions fit the best. 1, it could be added to the filesystem functions (we already have some which are only supported on some systems) 2, it could be added to the posix functions (would be logical) 3, it could be added as stream stuff (Gustavo supports this imo) 4, it could be added to the dio pecl extension (could fit in there, but it's not in the core, and the package doesn't seem much alive) 5, it could be added to pecl as a different extension (bad idea, imo) so, what do you think? Tyrael --- ## #52135 — Gustavo Lopes — 2011-05-08T02:21:02+00:00 URL: https://externals.io/message/52091#52135 · In reply to: #52134 > > > […] I just don't see the point of duplicating functionality, especially when there's already a closely related wrapper. It's not true that everything in http://php.net/wrappers can can be done in another fashion. In fact, for the php:// wrappers, which is what we're talking about, ONLY ONE (php://filter) duplicates functionality that can be achieved with functions. > > […] Yes, this causes a memory leak. Maybe that was the problem I found. Duping the file descriptor after the call to dup2 would solve this, which I think is one more reason not to expose these low-level functions directly and provide a safer alternative (more below). > > […] See http://lxr.php.net/xref/PHP_TRUNK/README.PARAMETER_PARSING_API#72 > and I forgot to ask the most important thing: > what do you think about the feature itself? > I would really like to have it in the core, but I think it would be > better to have this as functions, than adding more stream magic. > I'm not sure where would this functions fit the best. > 1, it could be added to the filesystem functions (we already have some > which are only supported on some systems) > 2, it could be added to the posix functions (would be logical) > 3, it could be added as stream stuff (Gustavo supports this imo) > 4, it could be added to the dio pecl extension (could fit in there, > but it's not in the core, and the package doesn't seem much alive) > 5, it could be added to pecl as a different extension (bad idea, imo) > PHP tries to abstract away these kind of details; I don't think direct access to low level I/O functions is appropriate in the core. The dio pecl extension would seem more appropriate. I don't know how alive it is, but if I recall correctly, it got a new maintainer last year. The POSIX extension would also seem to make sense, but it seems more focused on another class of functions and is not available on Windows. That said, if some useful use case is not available (or is available, but it's awkward) in PHP, we should try to fix it. If I understand correctly, what compels you is being able to replace stdin, stdout and stderr. I would tend to support #3; for reference here goes the change to php://fd that would be necessary. The problem is, as far as I know, there's no way to retrieve the file descriptor associated a stream, so we'd still have to rely on opening order. One possible solution would be to add this information to the wrapper data of the "plain files" and in systems other than Windows, to other relevant stream types (e.g. sockets). I think your solution of calling stream_cast is a good option because in Windows the socket streams return socket handles -- file descriptors are an abstraction in the C lib, not really part of the windows API -- and because in some stream implementations that call has important side effects. Another option would option would be to unburden the user from dealing with file descriptors completely and provide a function to replace the standard file descriptors with a stream. -- Gustavo Lopes --- ## #52136 — Ferenc Kovacs — 2011-05-08T09:23:59+00:00 URL: https://externals.io/message/52091#52136 · In reply to: #52135 > > adding dup2 support for the php://fd stream would be better than the > > […] > I just don't see the point of duplicating functionality, especially when > there's already a closely related wrapper. > > It's not true that everything in http://php.net/wrappers can can be done > in another fashion. In fact, for the php:// wrappers, which is what we're > talking about, ONLY ONE (php://filter) duplicates functionality that can be > achieved with functions. > > you are right, not everything, but many of them is either have alternative in core, or had it through pecl when that stream was introduced AFAIK . > > closing the stdout without reopening also caused problems for me in case > > […] > Yes, this causes a memory leak. Maybe that was the problem I found. Duping > the file descriptor after the call to dup2 would solve this, which I think > is one more reason not to expose these low-level functions directly and > provide a safer alternative (more below). > this is why I think that it would be better to provide a way to redirect/reopen the STDIN/STDOUT/STDERR. currently one can only close them, which could cause problmes, if something directly or indirectly tries to use one of those fps. > > > > […] > See http://lxr.php.net/xref/PHP_TRUNK/README.PARAMETER_PARSING_API#72 > > okay, I read that before, but somehow I belived that I should use ints because the C functions use its for fds. I will update that. > > and I forgot to ask the most important thing: > > […] > PHP tries to abstract away these kind of details; I don't think direct > access to low level I/O functions is appropriate in the core. The dio pecl > extension would seem more appropriate. I don't know how alive it is, but if > I recall correctly, it got a new maintainer last year. > > The POSIX extension would also seem to make sense, but it seems more > focused on another class of functions and is not available on Windows. > > That said, if some useful use case is not available (or is available, but > it's awkward) in PHP, we should try to fix it. If I understand correctly, > what compels you is being able to replace stdin, stdout and stderr. > I would tend to support #3; for reference here goes the change to php://fd > that would be necessary. The problem is, as far as I know, there's no way to > retrieve the file descriptor associated a stream, so we'd still have to rely > on opening order. > > One possible solution would be to add this information to the wrapper data > of the "plain files" and in systems other than Windows, to other relevant > stream types (e.g. sockets). I think your solution of calling stream_cast is > a good option because in Windows the socket streams return socket handles -- > file descriptors are an abstraction in the C lib, not really part of the > windows API -- and because in some stream implementations that call has > important side effects. > > Another option would option would be to unburden the user from dealing with > file descriptors completely and provide a function to replace the standard > file descriptors with a stream. > I agree, I think that the best solution would be to - add support for replacing a stream with another (a similar function for http://php.net/manual/en/function.stream-copy-to-stream.php or maybe extending that), which would be available on all platforms, this would cover my initial use-case. - I think we should also provide the fileno functionality, because without that, the php://fd is pretty useless. - anything else, like dup2, fcntl should go to the dio or posix extension, that would be useful for some people. what do you think? Tyrael --- ## #52137 — Ferenc Kovacs — 2011-05-08T13:51:01+00:00 URL: https://externals.io/message/52091#52137 · In reply to: #52136 > I agree, I think that the best solution would be to > - add support for replacing a stream with another (a similar function > for http://php.net/manual/en/function.stream-copy-to-stream.php or maybe > extending that), which would be available on all platforms, this would cover > my initial use-case. > - I think we should also provide the fileno functionality, because without > that, the php://fd is pretty useless. > - anything else, like dup2, fcntl should go to the dio or posix extension, > that would be useful for some people. > what do you think? > Tyrael the fileno could be added to the http://php.net/manual/en/function.stream-get-meta-data.php this way we still kept it in the stream functions without introducing a new function for it. Tyrael