# file streams & metadata - Thread: https://externals.io/message/51582 - Original archive: https://news-web.php.net/php.internals/51582 - Started by Stas Malyshev on 2011-03-07T01:15:47+00:00 - Messages: 30 - Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL) - Next page: https://externals.io/message/51582.md?page=2 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #51582 — Stas Malyshev — 2011-03-07T01:15:47+00:00 URL: https://externals.io/message/51582#51582 Hi! PHP file streams provide very powerful and useful abstraction layer over the I/O-related functions. However, there's a group of functions which are excluded from this support - namely, `touch()`, `chmod()`, `chown()` and `chgrp()` - i.e., functions dealing with file metadata. This lead to libraries implementing FS virtualization developing various hacks, such as here: https://code.google.com/p/bovigo/wiki/vfsStreamDocsFilemode I propose to plug this hole in the stream implementation by adding a handler to deal with metadata to stream handlers, such as: int (*stream_set_metadata)(php_stream_wrapper *wrapper, char *url, int metatype, void *metadata TSRMLS_DC); metadata types can be: - access/modification time with `touch()` semantics or without it - permissions information - owner information by name or number - group information by name or number Of course, there are streams where it doesn't make sense, but there are many application of streams (such as virtualizing real filesystems, mocks, etc.) where this would help a lot with using streams transparently inside applications. Any comments? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51589 — Gustavo Lopes — 2011-03-07T09:29:58+00:00 URL: https://externals.io/message/51582#51589 · In reply to: #51582 Em Mon, 07 Mar 2011 01:15:47 -0000, Stas Malyshev escreveu: > PHP file streams provide very powerful and useful abstraction layer over > the I/O-related functions. However, there's a group of functions which > are excluded from this support - namely, `touch()`, `chmod()`, `chown()` and > `chgrp()` - i.e., functions dealing with file metadata. This lead to > libraries implementing FS virtualization developing various hacks, such > as here: > https://code.google.com/p/bovigo/wiki/vfsStreamDocsFilemode > I propose to plug this hole in the stream implementation by adding a > handler to deal with metadata to stream handlers, such as: > > int (*stream_set_metadata)(php_stream_wrapper *wrapper, char *url, int > metatype, void *metadata TSRMLS_DC); > > metadata types can be: > - access/modification time with `touch()` semantics or without it > - permissions information > - owner information by name or number > - group information by name or number > > Of course, there are streams where it doesn't make sense, but there are > many application of streams (such as virtualizing real filesystems, > mocks, etc.) where this would help a lot with using streams > transparently inside applications. > Any comments? It's a good idea. But what about getting the metadata (as opposed to setting it)? I understand reading most of this stuff is already covered by the stat handler, but what about for example NTFS/POSIX ACLs or reading reparse point data? It would be nice to have a way to retrieve them. One might as well add two handlers. -- Gustavo Lopes --- ## #51590 — Pierre Joye — 2011-03-07T09:34:43+00:00 URL: https://externals.io/message/51582#51590 · In reply to: #51589 hi, > But what about getting the metadata (as opposed to setting it)? I understand > reading most of this stuff is already covered by the stat handler, but what > about for example NTFS/POSIX ACLs or reading reparse point data? It would be > nice to have a way to retrieve them. One might as well add two handlers. I would prefer to have these particular getters (and maybe ever setters) somewhere else, like in a derivated SplFile. That would give us more flexibility and cleaner APIs fro something that does not really fit in default streams anyway (or rarely). Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51592 — Stas Malyshev — 2011-03-07T09:44:29+00:00 URL: https://externals.io/message/51582#51592 · In reply to: #51590 Hi! > I would prefer to have these particular getters (and maybe ever > setters) somewhere else, like in a derivated SplFile. That would give > us more flexibility and cleaner APIs fro something that does not > really fit in default streams anyway (or rarely). `touch()`, `chmod()`, etc. aren't "rarely" used. They are used in alomst any app, everywhere. Right now every file wrapper - such as vfsStream mentioned before - is useless for transparent wrapping of the filesystem, since it can not cover these functions. And if you do an extension wrapping streams, you have to manually override these, which sucks big time, since it goes against the whole idea of streams abstraction. That means you can't test it, you can't use it for abstraction, nothing. Adding these functions into SplFile solves nothing - the whole point of streams is that they work with usual file functions, not require some separate class. If I needed separate class with its own functions, I won't be needing streams. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51593 — Pierre Joye — 2011-03-07T09:46:04+00:00 URL: https://externals.io/message/51582#51593 · In reply to: #51592 > Hi! > > > […] > > `touch()`, `chmod()`, etc. aren't "rarely" used. They are used in alomst any > app, everywhere. The key part of this reply was "these particular", as they are very platform specific and useless in a generic stream wrapper implementation. -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51595 — Stas Malyshev — 2011-03-07T09:54:04+00:00 URL: https://externals.io/message/51582#51595 · In reply to: #51593 Hi! > The key part of this reply was "these particular", as they are very > platform specific and useless in a generic stream wrapper > implementation. Every stream wrapper doesn't have to implement them - as it doesn't have to implement conversion to socket, etc. But without this, it is impossible to create a stream wrapper that implements full range of filesystem functions - which means, wrappers are useless for dealing with filesystem. This is a big hole in functionality, and it should be fixed. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51596 — Stas Malyshev — 2011-03-07T09:57:38+00:00 URL: https://externals.io/message/51582#51596 · In reply to: #51589 Hi! > But what about getting the metadata (as opposed to setting it)? I Most metadata widely used is covered by stat. I.e. for my itch (having functioning filesystem wrapper) it is not needed. But that doesn't mean it can't be done for other purposes :) -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51598 — Pierre Joye — 2011-03-07T10:11:00+00:00 URL: https://externals.io/message/51582#51598 · In reply to: #51595 > Hi! > > > […] > > Every stream wrapper doesn't have to implement them - as it doesn't have to > implement conversion to socket, etc. But without this, it is impossible to > create a stream wrapper that implements full range of filesystem functions - > which means, wrappers are useless for dealing with filesystem. This is a big > hole in functionality, and it should be fixed. To hear that stream are useless for filesystem operation is very new to me. I have to repeat what I said earlier, these specific informations (ACL, reparse points or other OS/FS specific informations) are very hard to implement in a generic way as you are proposing. As the idea in itself is good, I have serious doubts that a useful (working and portable) solution can be implemented with what is described in your initial post (barely posix only). -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51599 — Stas Malyshev — 2011-03-07T17:49:54+00:00 URL: https://externals.io/message/51582#51599 · In reply to: #51598 Hi! > To hear that stream are useless for filesystem operation is very new to me. It was surprise for me too, nevertheless once you try to use it, you pretty soon hit this roadblock with any serious application and have to resort to very ugly hacks. > > I have to repeat what I said earlier, these specific informations > (ACL, reparse points or other OS/FS specific informations) are very Nobody cares about reparse points. There's probably no PHP app in existence that cares about reparse points. There's tons of apps that use `touch()` and `chmod()` though. > hard to implement in a generic way as you are proposing. As the idea > in itself is good, I have serious doubts that a useful (working and > portable) solution can be implemented with what is described in your > initial post (barely posix only). It is useful, and it will be working. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51600 — Gustavo Lopes — 2011-03-07T18:12:35+00:00 URL: https://externals.io/message/51582#51600 · In reply to: #51599 Em Mon, 07 Mar 2011 17:49:54 -0000, Stas Malyshev escreveu: > > […] > > Nobody cares about reparse points. There's probably no PHP app in > existence that cares about reparse points. There's tons of apps that use > `touch()` and `chmod()` though. Even if one concedes "nobody/no PHP app cares about reparse points" (which is an odd statement for two reasons 1) reparse points are used to implement things such as symbolic links and 2) if there is no access to reparse points in PHP, saying no PHP app cares about them is a trivial necessity), are you seriously saying nobody cares about POSIX ACLs/extended attributes or ACLs/ownership information on Windows? This is a spectacularly selective reply... > > […] > > It is useful, and it will be working. There can be certainly no "working and portable" implementation of stuff like NTFS ACLs on *nix or a useful chmod implementation on Windows and platform specific implementations of those features in the standard stream wrappers will have to be ifdef'd out depending on the platform, and I think that's Pierre's concern. In this sense, it will be easier and more tempting to write platform-dependent PHP applications. However, while per se undesirable, I think it's less undesirable than limiting PHP to the lowest common denominator. -- Gustavo Lopes --- ## #51601 — Stas Malyshev — 2011-03-07T18:31:24+00:00 URL: https://externals.io/message/51582#51601 · In reply to: #51600 Hi! > There can be certainly no "working and portable" implementation of stuff > like NTFS ACLs on *nix or a useful chmod implementation on Windows and I am starting to feel you and Pierre are failing to see forest for the trees. The goal is not to implement every tiny detail of Windows FS functionality on any OS imaginable. The goal is to make common scenarios, used in myriads of existing applications, work with streams. If there's an app in PHP that does or needs NTFS ACLs, I don't address it. I address tons of other apps that just do `chmod()` and I want to make it work with streams the same it works without streams. I know it's not ideal and I don't want ideal - I want working. I don't want to design a perfect system that handles every imaginable case that never gets implemented because it will be bikeshedded to death on the list - I want simple and working solution that can be done by using existing functionality. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51602 — Pierre Joye — 2011-03-07T18:36:16+00:00 URL: https://externals.io/message/51582#51602 · In reply to: #51599 hi, > Hi! > > > […] > > It was surprise for me too, nevertheless once you try to use it, you pretty > soon hit this roadblock with any serious application and have to resort to > very ugly hacks. The more I read what you say the more I think that what you want is not a working stream implementation to do filesystem ops, but a way to emulate a virtual file system using stream. I don't think it is the goal of our stream implementation to do that. Tools like FuseFS are more appropriate and I can certainly do similar things on Windows as well. I have done that in the past for debugging purposes and we could certainly provide something as well. > > […] > > Nobody cares about reparse points. There's probably no PHP app in existence > that cares about reparse points. There's tons of apps that use `touch()` and > `chmod()` though. if nobody was taking care of them then PHP on windows will be miserable with junction, links and co already. But it is not. > > […] > > It is useful, and it will be working. I disagree for the reasons I explained earlier. It is even useless as it will be a very limited version. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51603 — Pierre Joye — 2011-03-07T18:37:30+00:00 URL: https://externals.io/message/51582#51603 · In reply to: #51601 > I am starting to feel you and Pierre are failing to see forest for the > trees. Well, actually I feel like you see one tree only, well two considering that you see Windows as being the only other tree. That's clearly not the case :) Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51604 — Stas Malyshev — 2011-03-07T19:45:38+00:00 URL: https://externals.io/message/51582#51604 · In reply to: #51602 Hi! > The more I read what you say the more I think that what you want is > not a working stream implementation to do filesystem ops, but a way to > emulate a virtual file system using stream. I don't think it is the > goal of our stream implementation to do that. Tools like FuseFS are I think our stream implementation is perfectly fine to do that, if one hole in it - the fact that some important functions completely ignore streams - is plugged. I don't need all-encompassing megapowerful super-portable virtual FS layer that would work in every OS imaginable and satisfy any scenario one could ever think of. I just need to be able to pass mystream://file to the app and be sure my app works. Right now I can't do that because `chmod()` and `touch()` break. Judging from vfsStream example (currently recommended solution for FS testing btw) I am not only one having this problem. If you have a solution that could do the same (namely, having PHP control access to files) in any other way - please tell me. But a practical way that can be done in reasonable time. > more appropriate and I can certainly do similar things on Windows as > well. I have done that in the past for debugging purposes and we could > certainly provide something as well. I'm not going to prevent you from writing a FUSE driver in PHP and also making it work on Linux, Windows, Mac, FreeBSD, etc. However, I have no idea how to do it and nobody else did it so far, AFAIK. I, however, have pretty good idea about how to make streams work, so I don't see how "let's design a new system 100% from scratch and implement it" is better than "let's fix existing and widely used system by adding missing 10% of functionality". If you can do better and implement your FUSE solution and it works better than extending streams - be my guest. > I disagree for the reasons I explained earlier. It is even useless as > it will be a very limited version. Did you just claim you know better than me what is useful for me? Nice :) But again vfsStream example shows it will be useful not only for myself but for many others. Your argument is that since it doesn't cover some fringe case which nobody currently uses and nobody ever asked for it's useless. Sorry, not buying it. It is no more limited than existing FS functions, and we're doing fine with them. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51605 — Pierre Joye — 2011-03-07T21:02:24+00:00 URL: https://externals.io/message/51582#51605 · In reply to: #51604 > Did you just claim you know better than me what is useful for me? Nice :) No I don't. But I'm telling you that what you propose is incomplete and bad, to start with. > But again vfsStream example shows it will be useful not only for myself but > for many others. Your argument is that since it doesn't cover some fringe > case which nobody currently uses You are not everybody, nor is your projects, sorry. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51606 — Stas Malyshev — 2011-03-07T21:09:43+00:00 URL: https://externals.io/message/51582#51606 · In reply to: #51605 Hi! > No I don't. But I'm telling you that what you propose is incomplete > and bad, to start with. Incomplete - yes. If we only implemented things that are "complete" in the sense you require, we would have no file operations in PHP at all - after all, we don't have support for reparse points and NTFS ACLs in file functions now, so by your logic they shouldn't be implemented until we design filesystem layer that allows to do it. Until then, PHP is better without any filesystem support at all. I don't see how it makes any sense. > > […] > > You are not everybody, nor is your projects, sorry. Come on, you know vfsStream is not my project and the problem is real. If you have proof that any of your counterexamples are widely used and streams are completely useless as filesystem layer without support for reparse points or NTFS ACLs - bring examples. Tell me which applications require them. Tell me how you unit-test them now and how you want to improve it. Without it, what I see is shooting down a good proposal because you can vaguely think of imagined better solution which nobody is going ever to design and implement. The result is we get no improvement at all. How is that any better? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51607 — Pierre Joye — 2011-03-07T21:12:11+00:00 URL: https://externals.io/message/51582#51607 · In reply to: #51606 > Incomplete - yes. If we only implemented things that are "complete" in the > sense you require, we would have no file operations in PHP at all - after > all, we don't have support for reparse points and NTFS ACLs in file > functions now, so by your logic they shouldn't be implemented until we > design filesystem layer that allows to do it. Until then, PHP is better > without any filesystem support at all. I don't see how it makes any sense. I do not see a sense to continue this discussion as you clearly ignore many aspects of the problem, even after we mentioned them numerous times in this thread. As in, it is not only about NTFS ACL or Windows only. Get over that first then it will be easier to actually discuss possible solutions or alternatives. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51608 — Stas Malyshev — 2011-03-07T21:17:39+00:00 URL: https://externals.io/message/51582#51608 · In reply to: #51607 Hi! > only. Get over that first then it will be easier to actually discuss > possible solutions or alternatives. I saw no possible solutions or alternatives mentioned except for vague references to FuseFS (which is totally different project, requires massive investment of time and would have same and worse compatibility problems) and mention of SplFile which is totally irrelevant. The only "aspect" so far was since it doesn't support 100% of every filesystem operations possible on any filesystem in existence we shouldn't provide support for even the most common of them. Which makes no sense since it's exactly what we are doing now and always did in PHP. I only propose to extend it a bit to support one more, to get better coverage. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51609 — Pierre Joye — 2011-03-07T21:25:08+00:00 URL: https://externals.io/message/51582#51609 · In reply to: #51608 > Hi! > > > […] > > I saw no possible solutions or alternatives mentioned except for vague > references to FuseFS (which is totally different project, requires massive > investment of time and would have same and worse compatibility problems) and > mention of SplFile which is totally irrelevant. It is irritating that you don't actually read the replies but reply as fast as we were on IRC. I mentioned SplFile for the specific informations mentioned by Gustavo, the ACL, reparse data or related informations (windows specific or not, that is not the point). As these informations are very operating systems specific and in many cases, even file system specific. That's not what you want to hear, but they are facts. That was also the reasons why they are not present in our stream implementation. I could image something for touch (there is a FR for it afair) as it is very easy to emulate on stream where it could not work, but really not for ch*. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51610 — Stas Malyshev — 2011-03-07T21:32:29+00:00 URL: https://externals.io/message/51582#51610 · In reply to: #51609 Hi! > That was also the reasons why they are not present in our stream > implementation. I could image something for touch (there is a FR for > it afair) as it is very easy to emulate on stream where it could not > work, but really not for ch*. We have chmod now defined on all systems. However good or bad it is, I see no problems in doing the same for streams. Streams that don't have this semantics are free to not define this handler or not to support this particular option, as happens with other optional handlers - e.g., mkdir makes no sense for sockets, but that doesn't prevent mkdir from existing in streams or socket streams from working fine. If you look at the docs, you see there's even a section for streams saying what's supported and what's not by particular stream, so obviously streams not supporting some capabilities always were fine. So where's the problem? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51611 — Ferenc Kovacs — 2011-03-07T21:49:08+00:00 URL: https://externals.io/message/51582#51611 · In reply to: #51610 On Mon, Mar 7, 2011 at 10:32 PM, Stas Malyshev wrote: > Hi! > > > That was also the reasons why they are not present in our stream > > […] > > We have chmod now defined on all systems. However good or bad it is, I see > no problems in doing the same for streams. Streams that don't have this > semantics are free to not define this handler or not to support this > particular option, as happens with other optional handlers - e.g., mkdir > makes no sense for sockets, but that doesn't prevent mkdir from existing in > streams or socket streams from working fine. If you look at the docs, you > see there's even a section for streams saying what's supported and what's > not by particular stream, so obviously streams not supporting some > capabilities always were fine. So where's the problem? > > -- > Stanislav Malyshev, Software Architect > SugarCRM: http://www.sugarcrm.com/ > (408)454-6900 ext. 227 > > -- > > > > From the sideline: it's hard to belive that there are no one else with opinion on this matter. I don't want to take sides because I don't have the necessary knowledge about the matter, but maybe it would be a good idea to write an RFC, and and maybe a POC, so we can see how intrusive would be the implementation. and if there are more bystanders, then please make your voice heard. Tyrael --- ## #51612 — Stas Malyshev — 2011-03-07T21:53:46+00:00 URL: https://externals.io/message/51582#51612 · In reply to: #51611 Hi! > it's hard to belive that there are no one else with opinion on this matter. > I don't want to take sides because I don't have the necessary knowledge > about the matter, but maybe it would be a good idea to write an RFC, and > and maybe a POC, so we can see how intrusive would be the implementation. > and if there are more bystanders, then please make your voice heard. I intend to do both soon. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 --- ## #51613 — Pierre Joye — 2011-03-08T12:55:46+00:00 URL: https://externals.io/message/51582#51613 · In reply to: #51611 > From the sideline: > it's hard to belive that there are no one else with opinion on this matter. > I don't want to take sides because I don't have the necessary knowledge > about the matter, but maybe it would be a good idea to write an RFC, and and > maybe a POC, so we can see how intrusive would be the implementation. > and if there are more bystanders, then please make your voice heard. > Tyrael It is very easy to implement what Stas is asking. But that's not the question. The question is should we do it? And my point is that we should not do it like that, bad design, incomplete and limited support. -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- ## #51617 — Matthew Weier O'Phinney — 2011-03-08T15:46:15+00:00 URL: https://externals.io/message/51582#51617 · In reply to: #51613 > > > > […] > > It is very easy to implement what Stas is asking. But that's not the > question. The question is should we do it? And my point is that we > should not do it like that, bad design, incomplete and limited > support. As a developer, in many cases I'd prefer something incomplete and limited in support to nothing at all. I've been bitten by the same issue Stas describes, and having a way to tie into touch and chmod in my stream wrappers would be incredibly useful -- even if it won't work with _every_ stream. -- Matthew Weier O'Phinney Project Lead | matthew@zend.com Zend Framework | http://framework.zend.com/ PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc --- ## #51618 — Pierre Joye — 2011-03-08T15:52:13+00:00 URL: https://externals.io/message/51582#51618 · In reply to: #51617 On Tue, Mar 8, 2011 at 4:46 PM, Matthew Weier O'Phinney wrote: > As a developer, in many cases I'd prefer something incomplete and limited in > support to nothing at all. I've been bitten by the same issue Stas describes, > and having a way to tie into touch and chmod in my stream wrappers would be > incredibly useful -- even if it won't work with _every_ stream. I'm not saying that this feature is not necessary. Only that what is proposed will hit us hardly very soon after we have released it. And we will have to live and hack with it for a long time. That's a price I don't want to pay. All I'm saying is to actually sit down and think about what we are doing instead of running into the 1st solution, as easy as it could be. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- Next page: https://externals.io/message/51582.md?page=2