Hi!
PHP file streams provide very powerful and useful abstraction layer over
the I/O-related functions. However, there's a group of functions which
are excluded from this support - namely, touch(), chmod(), chown() and
chgrp() - i.e., functions dealing with file metadata. This lead to
libraries implementing FS virtualization developing various hacks, such
as here:
https://code.google.com/p/bovigo/wiki/vfsStreamDocsFilemode
I propose to plug this hole in the stream implementation by adding a
handler to deal with metadata to stream handlers, such as:
int (*stream_set_metadata)(php_stream_wrapper *wrapper, char *url, int
metatype, void *metadata TSRMLS_DC);
metadata types can be:
- access/modification time with
touch()semantics or without it - permissions information
- owner information by name or number
- group information by name or number
Of course, there are streams where it doesn't make sense, but there are
many application of streams (such as virtualizing real filesystems,
mocks, etc.) where this would help a lot with using streams
transparently inside applications.
Any comments?
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Em Mon, 07 Mar 2011 01:15:47 -0000, Stas Malyshev smalyshev@sugarcrm.com
escreveu:
PHP file streams provide very powerful and useful abstraction layer over
the I/O-related functions. However, there's a group of functions which
are excluded from this support - namely,touch(),chmod(),chown()and
chgrp()- i.e., functions dealing with file metadata. This lead to
libraries implementing FS virtualization developing various hacks, such
as here:
https://code.google.com/p/bovigo/wiki/vfsStreamDocsFilemode
I propose to plug this hole in the stream implementation by adding a
handler to deal with metadata to stream handlers, such as:int (*stream_set_metadata)(php_stream_wrapper *wrapper, char *url, int
metatype, void *metadata TSRMLS_DC);metadata types can be:
- access/modification time with
touch()semantics or without it- permissions information
- owner information by name or number
- group information by name or number
Of course, there are streams where it doesn't make sense, but there are
many application of streams (such as virtualizing real filesystems,
mocks, etc.) where this would help a lot with using streams
transparently inside applications.
Any comments?
It's a good idea.
But what about getting the metadata (as opposed to setting it)? I
understand reading most of this stuff is already covered by the stat
handler, but what about for example NTFS/POSIX ACLs or reading reparse
point data? It would be nice to have a way to retrieve them. One might as
well add two handlers.
--
Gustavo Lopes
hi,
But what about getting the metadata (as opposed to setting it)? I understand
reading most of this stuff is already covered by the stat handler, but what
about for example NTFS/POSIX ACLs or reading reparse point data? It would be
nice to have a way to retrieve them. One might as well add two handlers.
I would prefer to have these particular getters (and maybe ever
setters) somewhere else, like in a derivated SplFile. That would give
us more flexibility and cleaner APIs fro something that does not
really fit in default streams anyway (or rarely).
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
I would prefer to have these particular getters (and maybe ever
setters) somewhere else, like in a derivated SplFile. That would give
us more flexibility and cleaner APIs fro something that does not
really fit in default streams anyway (or rarely).
touch(), chmod(), etc. aren't "rarely" used. They are used in alomst any
app, everywhere. Right now every file wrapper - such as vfsStream
mentioned before - is useless for transparent wrapping of the
filesystem, since it can not cover these functions. And if you do an
extension wrapping streams, you have to manually override these, which
sucks big time, since it goes against the whole idea of streams abstraction.
That means you can't test it, you can't use it for abstraction, nothing.
Adding these functions into SplFile solves nothing - the whole point of
streams is that they work with usual file functions, not require some
separate class. If I needed separate class with its own functions, I
won't be needing streams.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
I would prefer to have these particular getters (and maybe ever
setters) somewhere else, like in a derivated SplFile. That would give
us more flexibility and cleaner APIs fro something that does not
really fit in default streams anyway (or rarely).
touch(),chmod(), etc. aren't "rarely" used. They are used in alomst any
app, everywhere.
The key part of this reply was "these particular", as they are very
platform specific and useless in a generic stream wrapper
implementation.
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
The key part of this reply was "these particular", as they are very
platform specific and useless in a generic stream wrapper
implementation.
Every stream wrapper doesn't have to implement them - as it doesn't have
to implement conversion to socket, etc. But without this, it is
impossible to create a stream wrapper that implements full range of
filesystem functions - which means, wrappers are useless for dealing
with filesystem. This is a big hole in functionality, and it should be
fixed.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
The key part of this reply was "these particular", as they are very
platform specific and useless in a generic stream wrapper
implementation.Every stream wrapper doesn't have to implement them - as it doesn't have to
implement conversion to socket, etc. But without this, it is impossible to
create a stream wrapper that implements full range of filesystem functions -
which means, wrappers are useless for dealing with filesystem. This is a big
hole in functionality, and it should be fixed.
To hear that stream are useless for filesystem operation is very new to me.
I have to repeat what I said earlier, these specific informations
(ACL, reparse points or other OS/FS specific informations) are very
hard to implement in a generic way as you are proposing. As the idea
in itself is good, I have serious doubts that a useful (working and
portable) solution can be implemented with what is described in your
initial post (barely posix only).
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
To hear that stream are useless for filesystem operation is very new to me.
It was surprise for me too, nevertheless once you try to use it, you
pretty soon hit this roadblock with any serious application and have to
resort to very ugly hacks.
I have to repeat what I said earlier, these specific informations
(ACL, reparse points or other OS/FS specific informations) are very
Nobody cares about reparse points. There's probably no PHP app in
existence that cares about reparse points. There's tons of apps that use
touch() and chmod() though.
hard to implement in a generic way as you are proposing. As the idea
in itself is good, I have serious doubts that a useful (working and
portable) solution can be implemented with what is described in your
initial post (barely posix only).
It is useful, and it will be working.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Em Mon, 07 Mar 2011 17:49:54 -0000, Stas Malyshev smalyshev@sugarcrm.com
escreveu:
I have to repeat what I said earlier, these specific informations
(ACL, reparse points or other OS/FS specific informations) are veryNobody cares about reparse points. There's probably no PHP app in
existence that cares about reparse points. There's tons of apps that use
touch()andchmod()though.
Even if one concedes "nobody/no PHP app cares about reparse points" (which
is an odd statement for two reasons 1) reparse points are used to
implement things such as symbolic links and 2) if there is no access to
reparse points in PHP, saying no PHP app cares about them is a trivial
necessity), are you seriously saying nobody cares about POSIX
ACLs/extended attributes or ACLs/ownership information on Windows? This is
a spectacularly selective reply...
hard to implement in a generic way as you are proposing. As the idea
in itself is good, I have serious doubts that a useful (working and
portable) solution can be implemented with what is described in your
initial post (barely posix only).It is useful, and it will be working.
There can be certainly no "working and portable" implementation of stuff
like NTFS ACLs on *nix or a useful chmod implementation on Windows and
platform specific implementations of those features in the standard stream
wrappers will have to be ifdef'd out depending on the platform, and I
think that's Pierre's concern.
In this sense, it will be easier and more tempting to write
platform-dependent PHP applications. However, while per se undesirable, I
think it's less undesirable than limiting PHP to the lowest common
denominator.
--
Gustavo Lopes
Hi!
There can be certainly no "working and portable" implementation of stuff
like NTFS ACLs on *nix or a useful chmod implementation on Windows and
I am starting to feel you and Pierre are failing to see forest for the
trees. The goal is not to implement every tiny detail of Windows FS
functionality on any OS imaginable. The goal is to make common
scenarios, used in myriads of existing applications, work with streams.
If there's an app in PHP that does or needs NTFS ACLs, I don't address
it. I address tons of other apps that just do chmod() and I want to make
it work with streams the same it works without streams. I know it's not
ideal and I don't want ideal - I want working. I don't want to design a
perfect system that handles every imaginable case that never gets
implemented because it will be bikeshedded to death on the list - I want
simple and working solution that can be done by using existing
functionality.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
I am starting to feel you and Pierre are failing to see forest for the
trees.
Well, actually I feel like you see one tree only, well two considering
that you see Windows as being the only other tree. That's clearly not
the case :)
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
hi,
Hi!
To hear that stream are useless for filesystem operation is very new to
me.It was surprise for me too, nevertheless once you try to use it, you pretty
soon hit this roadblock with any serious application and have to resort to
very ugly hacks.
The more I read what you say the more I think that what you want is
not a working stream implementation to do filesystem ops, but a way to
emulate a virtual file system using stream. I don't think it is the
goal of our stream implementation to do that. Tools like FuseFS are
more appropriate and I can certainly do similar things on Windows as
well. I have done that in the past for debugging purposes and we could
certainly provide something as well.
I have to repeat what I said earlier, these specific informations
(ACL, reparse points or other OS/FS specific informations) are veryNobody cares about reparse points. There's probably no PHP app in existence
that cares about reparse points. There's tons of apps that usetouch()and
chmod()though.
if nobody was taking care of them then PHP on windows will be
miserable with junction, links and co already. But it is not.
hard to implement in a generic way as you are proposing. As the idea
in itself is good, I have serious doubts that a useful (working and
portable) solution can be implemented with what is described in your
initial post (barely posix only).It is useful, and it will be working.
I disagree for the reasons I explained earlier. It is even useless as
it will be a very limited version.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
The more I read what you say the more I think that what you want is
not a working stream implementation to do filesystem ops, but a way to
emulate a virtual file system using stream. I don't think it is the
goal of our stream implementation to do that. Tools like FuseFS are
I think our stream implementation is perfectly fine to do that, if one
hole in it - the fact that some important functions completely ignore
streams - is plugged. I don't need all-encompassing megapowerful
super-portable virtual FS layer that would work in every OS imaginable
and satisfy any scenario one could ever think of. I just need to be able
to pass mystream://file to the app and be sure my app works. Right now I
can't do that because chmod() and touch() break. Judging from vfsStream
example (currently recommended solution for FS testing btw) I am not
only one having this problem. If you have a solution that could do the
same (namely, having PHP control access to files) in any other way -
please tell me. But a practical way that can be done in reasonable time.
more appropriate and I can certainly do similar things on Windows as
well. I have done that in the past for debugging purposes and we could
certainly provide something as well.
I'm not going to prevent you from writing a FUSE driver in PHP and also
making it work on Linux, Windows, Mac, FreeBSD, etc. However, I have no
idea how to do it and nobody else did it so far, AFAIK. I, however, have
pretty good idea about how to make streams work, so I don't see how
"let's design a new system 100% from scratch and implement it" is better
than "let's fix existing and widely used system by adding missing 10% of
functionality".
If you can do better and implement your FUSE solution and it works
better than extending streams - be my guest.
I disagree for the reasons I explained earlier. It is even useless as
it will be a very limited version.
Did you just claim you know better than me what is useful for me? Nice
:) But again vfsStream example shows it will be useful not only for
myself but for many others. Your argument is that since it doesn't cover
some fringe case which nobody currently uses and nobody ever asked for
it's useless. Sorry, not buying it. It is no more limited than existing
FS functions, and we're doing fine with them.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Did you just claim you know better than me what is useful for me? Nice :)
No I don't. But I'm telling you that what you propose is incomplete
and bad, to start with.
But again vfsStream example shows it will be useful not only for myself but
for many others. Your argument is that since it doesn't cover some fringe
case which nobody currently uses
You are not everybody, nor is your projects, sorry.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
No I don't. But I'm telling you that what you propose is incomplete
and bad, to start with.
Incomplete - yes. If we only implemented things that are "complete" in
the sense you require, we would have no file operations in PHP at all -
after all, we don't have support for reparse points and NTFS ACLs in
file functions now, so by your logic they shouldn't be implemented until
we design filesystem layer that allows to do it. Until then, PHP is
better without any filesystem support at all. I don't see how it makes
any sense.
But again vfsStream example shows it will be useful not only for myself but
for many others. Your argument is that since it doesn't cover some fringe
case which nobody currently usesYou are not everybody, nor is your projects, sorry.
Come on, you know vfsStream is not my project and the problem is real.
If you have proof that any of your counterexamples are widely used and
streams are completely useless as filesystem layer without support for
reparse points or NTFS ACLs - bring examples. Tell me which applications
require them. Tell me how you unit-test them now and how you want to
improve it. Without it, what I see is shooting down a good proposal
because you can vaguely think of imagined better solution which nobody
is going ever to design and implement. The result is we get no
improvement at all. How is that any better?
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Incomplete - yes. If we only implemented things that are "complete" in the
sense you require, we would have no file operations in PHP at all - after
all, we don't have support for reparse points and NTFS ACLs in file
functions now, so by your logic they shouldn't be implemented until we
design filesystem layer that allows to do it. Until then, PHP is better
without any filesystem support at all. I don't see how it makes any sense.
I do not see a sense to continue this discussion as you clearly ignore
many aspects of the problem, even after we mentioned them numerous
times in this thread. As in, it is not only about NTFS ACL or Windows
only. Get over that first then it will be easier to actually discuss
possible solutions or alternatives.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
only. Get over that first then it will be easier to actually discuss
possible solutions or alternatives.
I saw no possible solutions or alternatives mentioned except for vague
references to FuseFS (which is totally different project, requires
massive investment of time and would have same and worse compatibility
problems) and mention of SplFile which is totally irrelevant.
The only "aspect" so far was since it doesn't support 100% of every
filesystem operations possible on any filesystem in existence we
shouldn't provide support for even the most common of them. Which makes
no sense since it's exactly what we are doing now and always did in PHP.
I only propose to extend it a bit to support one more, to get better
coverage.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
only. Get over that first then it will be easier to actually discuss
possible solutions or alternatives.I saw no possible solutions or alternatives mentioned except for vague
references to FuseFS (which is totally different project, requires massive
investment of time and would have same and worse compatibility problems) and
mention of SplFile which is totally irrelevant.
It is irritating that you don't actually read the replies but reply as
fast as we were on IRC.
I mentioned SplFile for the specific informations mentioned by
Gustavo, the ACL, reparse data or related informations (windows
specific or not, that is not the point). As these informations are
very operating systems specific and in many cases, even file system
specific. That's not what you want to hear, but they are facts.
That was also the reasons why they are not present in our stream
implementation. I could image something for touch (there is a FR for
it afair) as it is very easy to emulate on stream where it could not
work, but really not for ch*.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Hi!
That was also the reasons why they are not present in our stream
implementation. I could image something for touch (there is a FR for
it afair) as it is very easy to emulate on stream where it could not
work, but really not for ch*.
We have chmod now defined on all systems. However good or bad it is, I
see no problems in doing the same for streams. Streams that don't have
this semantics are free to not define this handler or not to support
this particular option, as happens with other optional handlers - e.g.,
mkdir makes no sense for sockets, but that doesn't prevent mkdir from
existing in streams or socket streams from working fine. If you look at
the docs, you see there's even a section for streams saying what's
supported and what's not by particular stream, so obviously streams not
supporting some capabilities always were fine. So where's the problem?
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
On Mon, Mar 7, 2011 at 10:32 PM, Stas Malyshev smalyshev@sugarcrm.comwrote:
Hi!
That was also the reasons why they are not present in our stream
implementation. I could image something for touch (there is a FR for
it afair) as it is very easy to emulate on stream where it could not
work, but really not for ch*.We have chmod now defined on all systems. However good or bad it is, I see
no problems in doing the same for streams. Streams that don't have this
semantics are free to not define this handler or not to support this
particular option, as happens with other optional handlers - e.g., mkdir
makes no sense for sockets, but that doesn't prevent mkdir from existing in
streams or socket streams from working fine. If you look at the docs, you
see there's even a section for streams saying what's supported and what's
not by particular stream, so obviously streams not supporting some
capabilities always were fine. So where's the problem?--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227--
From the sideline:
it's hard to belive that there are no one else with opinion on this matter.
I don't want to take sides because I don't have the necessary knowledge
about the matter, but maybe it would be a good idea to write an RFC, and and
maybe a POC, so we can see how intrusive would be the implementation.
and if there are more bystanders, then please make your voice heard.
Tyrael
Hi!
it's hard to belive that there are no one else with opinion on this matter.
I don't want to take sides because I don't have the necessary knowledge
about the matter, but maybe it would be a good idea to write an RFC, and
and maybe a POC, so we can see how intrusive would be the implementation.
and if there are more bystanders, then please make your voice heard.
I intend to do both soon.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
From the sideline:
it's hard to belive that there are no one else with opinion on this matter.
I don't want to take sides because I don't have the necessary knowledge
about the matter, but maybe it would be a good idea to write an RFC, and and
maybe a POC, so we can see how intrusive would be the implementation.
and if there are more bystanders, then please make your voice heard.
Tyrael
It is very easy to implement what Stas is asking. But that's not the
question. The question is should we do it? And my point is that we
should not do it like that, bad design, incomplete and limited
support.
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
From the sideline:
it's hard to belive that there are no one else with opinion on this matter.
I don't want to take sides because I don't have the necessary knowledge
about the matter, but maybe it would be a good idea to write an RFC, and and
maybe a POC, so we can see how intrusive would be the implementation.
and if there are more bystanders, then please make your voice heard.
TyraelIt is very easy to implement what Stas is asking. But that's not the
question. The question is should we do it? And my point is that we
should not do it like that, bad design, incomplete and limited
support.
As a developer, in many cases I'd prefer something incomplete and limited in
support to nothing at all. I've been bitten by the same issue Stas describes,
and having a way to tie into touch and chmod in my stream wrappers would be
incredibly useful -- even if it won't work with every stream.
--
Matthew Weier O'Phinney
Project Lead | matthew@zend.com
Zend Framework | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
On Tue, Mar 8, 2011 at 4:46 PM, Matthew Weier O'Phinney
weierophinney@php.net wrote:
As a developer, in many cases I'd prefer something incomplete and limited in
support to nothing at all. I've been bitten by the same issue Stas describes,
and having a way to tie into touch and chmod in my stream wrappers would be
incredibly useful -- even if it won't work with every stream.
I'm not saying that this feature is not necessary. Only that what is
proposed will hit us hardly very soon after we have released it. And
we will have to live and hack with it for a long time. That's a price
I don't want to pay.
All I'm saying is to actually sit down and think about what we are
doing instead of running into the 1st solution, as easy as it could
be.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
On Tue, Mar 8, 2011 at 4:46 PM, Matthew Weier O'Phinney
weierophinney@php.net wrote:
and having a way to tie into touch and chmod in my stream wrappers would be
incredibly useful -- even if it won't work with every stream.
My point was not that it won't work with any stream, obviously. But
that it won't work on many systems (various OS and filesystems
combinations).
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
From the sideline:
it's hard to belive that there are no one else with opinion on this
matter.
I don't want to take sides because I don't have the necessary knowledge
about the matter, but maybe it would be a good idea to write an RFC, and
and
maybe a POC, so we can see how intrusive would be the implementation.
and if there are more bystanders, then please make your voice heard.
TyraelIt is very easy to implement what Stas is asking. But that's not the
question. The question is should we do it? And my point is that we
should not do it like that, bad design, incomplete and limited
support.
if you have a better alternative which is viable, then I would support that.
otherwise I wouldn't support the all-or-nothing approach.
thats like if we couldn't add pecl/phpize until it works on every supported
platform/filesystem combination. (cheers for the upcoming 5.3.6 release)
and there is a ton of stuff which some point in time (or still ) didn't work
on all platform.
I think the best thing would be extending streams for every possible
method/operation, but require the userland developer to register callbacks
for the requested operation, and this way vfsStream devs could mock the
chmod calls but the built-in wrappers still would behave correctly (if they
didn't explicitly implement/support that operation, they would throw an
warning just like now)
it's not a generic implementation, but we could allow every single method to
be mockable, and we could gradually add support for the new operations for
the built-in wrappers.
maybe there are other reasons/use-cases for which this solution is not good
enough, but it would solve the the initial problem.
ps: sorry if I'm missing something, please feel free to point that out.
Tyrael
if you have a better alternative which is viable, then I would support that.
otherwise I wouldn't support the all-or-nothing approach.
thats like if we couldn't add pecl/phpize until it works on every supported
platform/filesystem combination. (cheers for the upcoming 5.3.6 release)
and there is a ton of stuff which some point in time (or still ) didn't work
on all platform.
No, that's a totally different problem. phpize has no impact on the
internal APIs and how things work in the engine or stream.
Making bad things stream (and god knows how many of these things we
already have) has a direct impact on every platform we support. Here
it will be even worst as it will be on every platform+FS combination.
It is pretty much the same as doing drivers specific things as the PDO
level and make them look like being generic. Very very bad idea,
obviously.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
if you have a better alternative which is viable, then I would support
that.
otherwise I wouldn't support the all-or-nothing approach.
thats like if we couldn't add pecl/phpize until it works on every
supported
platform/filesystem combination. (cheers for the upcoming 5.3.6 release)
and there is a ton of stuff which some point in time (or still ) didn't
work
on all platform.No, that's a totally different problem. phpize has no impact on the
internal APIs and how things work in the engine or stream.Making bad things stream (and god knows how many of these things we
already have) has a direct impact on every platform we support. Here
it will be even worst as it will be on every platform+FS combination.It is pretty much the same as doing drivers specific things as the PDO
level and make them look like being generic. Very very bad idea,
obviously.
while I agree that we shouldn't carelessly add something that will be a PITA
to support later, but I think that we should watch out for the
over-engineering/Nirvana fallacy also.
this is why I said that it would be a good idea to write things down, and
get more brains and yes on the problem and the suggested solutions.
and I think that with the last emails, we've started to get constructive
about the matter.
Tyrael
Hi!
But what about getting the metadata (as opposed to setting it)? I
Most metadata widely used is covered by stat. I.e. for my itch (having
functioning filesystem wrapper) it is not needed. But that doesn't mean
it can't be done for other purposes :)
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
Here's an RFC:
http://wiki.php.net/rfc/streammetadata
Patch link inside.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227