# memory_get_usage with new Memory Manager - Thread: https://externals.io/message/24954 - Original archive: https://news-web.php.net/php.internals/24954 - Started by Matt W on 2006-07-25T03:52:31+00:00 - Messages: 16 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #24954 — Matt W — 2006-07-25T03:52:31+00:00 URL: https://externals.io/message/24954#24954 Hi, After looking through the new Memory Manager code, I assumed this would be the case... (and just now got to test). Since heap->size is only updated when a 256K or whatever block is *actually* allocated/freed, the number returned by `memory_get_usage()` will only be a multiple of that. That doesn't make it very useful anymore if one is using it to see how much memory usage differs by changing, say, number of arrays/elements/variables, strings, etc. as I was last month (small changes). And the example in the manual definitely won't work as shown. Does anybody else think that memory_get[peak_]usage() should work more like before? I would imagine it can be "fixed" fairly easily, though it probably needs another variable, and just a little extra overhead of inc./dec. that variable more often (but I think that was done in the old MM anyway?). Thanks, Matt --- ## #24955 — Dmitry Stogov — 2006-07-25T06:13:48+00:00 URL: https://externals.io/message/24954#24955 · In reply to: #24954 Right now memory_get[peak_]usage() show the amount of REAL memory that PHP (Zend Memory Manager) takes from system. Previous memory manager showed size of emalloc()-ed memory without malloc() overhead. Also it didn't consider internal caches. We need decide which behavior to use before 5.2.0 release. Thanks. Dmitry. > -----Original Message----- > From: Matt W [mailto:php_lists@realplain.com] > Sent: Tuesday, July 25, 2006 7:53 AM > To: internals@lists.php.net > Subject: [PHP-DEV] memory_get_usage with new Memory Manager > > > Hi, > > After looking through the new Memory Manager code, I assumed > this would be the case... (and just now got to test). Since > heap->size is only updated when a 256K or whatever block is > *actually* allocated/freed, the number returned by > `memory_get_usage()` will only be a multiple of that. That > doesn't make it very useful anymore if one is using it to see > how much memory usage differs by changing, say, number of > arrays/elements/variables, strings, etc. as I was last month > (small changes). And the example in the manual definitely > won't work as shown. > > Does anybody else think that memory_get[peak_]usage() should > work more like before? I would imagine it can be "fixed" > fairly easily, though it probably needs another variable, and > just a little extra overhead of inc./dec. that variable more > often (but I think that was done in the old MM anyway?). > > > Thanks, > Matt --- ## #24965 — Ilia Alshanetsky — 2006-07-25T12:07:05+00:00 URL: https://externals.io/message/24954#24965 · In reply to: #24954 > Does anybody else think that memory_get[peak_]usage() should work > more like > before? I would imagine it can be "fixed" fairly easily, though it > probably > needs another variable, and just a little extra overhead of inc./ > dec. that > variable more often (but I think that was done in the old MM anyway?). I think that the current, more precise memory usage indicator is the correct behavior and should not be changed. Ilia Alshanetsky --- ## #24968 — Lukas Smith — 2006-07-25T13:45:03+00:00 URL: https://externals.io/message/24954#24968 · In reply to: #24955 Dmitry Stogov wrote: > Right now memory_get[peak_]usage() show the amount of REAL memory that PHP > (Zend Memory Manager) takes from system. > > Previous memory manager showed size of emalloc()-ed memory without malloc() > overhead. > Also it didn't consider internal caches. Shouldn't we make the old behavior the default here? regards, Lukas --- ## #24970 — Ilia Alshanetsky — 2006-07-25T14:10:12+00:00 URL: https://externals.io/message/24954#24970 · In reply to: #24968 The idea of a memory limit is to as accurately as possible account for the memory utilized by PHP. If our current calculation is more accurate since it accounts for malloc overhead, great, it gives more fine grained control to hosters utilizing this option. > Dmitry Stogov wrote: > > […] > > Shouldn't we make the old behavior the default here? > > regards, > Lukas > > -- > > > > Ilia Alshanetsky --- ## #24969 — Antony Dovgal — 2006-07-25T14:48:57+00:00 URL: https://externals.io/message/24954#24969 · In reply to: #24968 > Dmitry Stogov wrote: > > […] > > Shouldn't we make the old behavior the default here? Agree. -- Wbr, Antony Dovgal --- ## #24995 — Matt W — 2006-07-27T10:02:26+00:00 URL: https://externals.io/message/24954#24995 · In reply to: #24970 Hi Dmitry, Ilia, et al., Ilia, the Memory Manager is checking the *real* size against memory_limit, so it's still "more accurate" even after the functions were changed to report more like the old way. I still wonder how much difference there may be between size and real_size as it gets close to the limit... After the changes, I see the usage difference after adding a variable, for example, is slightly different than before, but I understand that's because of counting overhead now or such. The old one just rounded the amount up to a multiple of 8, I think... Something else I'd like to see changed... Does anyone think memory_get_[peak_]usage() should *always* be available, regardless of whether memory_limit is actually enabled? Using the Windows binaries at least, I was devastated :-D that they were unavailable for experimenting, etc. I don't see why not -- the function names IMO don't really suggest that they're tied to memory_limit and all that's being saved by not having them is a little overhead of updating the size/peak variables, right? Thanks, Matt ----- Original Message ----- From: "Ilia Alshanetsky" Sent: Tuesday, July 25, 2006 > The idea of a memory limit is to as accurately as possible account > for the memory utilized by PHP. If our current calculation is more > accurate since it accounts for malloc overhead, great, it gives more > fine grained control to hosters utilizing this option. > > > > > > […] --- ## #24996 — Edin Kadribasic — 2006-07-27T10:39:47+00:00 URL: https://externals.io/message/24954#24996 · In reply to: #24995 Matt W wrote: > Hi Dmitry, Ilia, et al., > > Ilia, the Memory Manager is checking the *real* size against memory_limit, > so it's still "more accurate" even after the functions were changed to > report more like the old way. I still wonder how much difference there may > be between size and real_size as it gets close to the limit... > > After the changes, I see the usage difference after adding a variable, for > example, is slightly different than before, but I understand that's because > of counting overhead now or such. The old one just rounded the amount up to > a multiple of 8, I think... > > Something else I'd like to see changed... Does anyone think > memory_get_[peak_]usage() should *always* be available, regardless of > whether memory_limit is actually enabled? Using the Windows binaries at > least, I was devastated :-D that they were unavailable for experimenting, > etc. > > I don't see why not -- the function names IMO don't really suggest that > they're tied to memory_limit and all that's being saved by not having them > is a little overhead of updating the size/peak variables, right? > The reason is performance. It takes some processing time to keep accounting of memory usage and people don't using the feature do not feel like its worth slowing PHP down. Edin --- ## #25000 — Ilia Alshanetsky — 2006-07-27T12:34:47+00:00 URL: https://externals.io/message/24954#25000 · In reply to: #24995 > > Something else I'd like to see changed... Does anyone think > memory_get_[peak_]usage() should *always* be available, regardless of > whether memory_limit is actually enabled? Using the Windows > binaries at > least, I was devastated :-D that they were unavailable for > experimenting, > etc. > Counting memory all the time adds a fair amount of overhead to script execution especially given how frequently PHP needs to allocate memory. Therefore usage memory utilization functions are only available when PHP is compiled with memory in which case it keeps track of its mem usage. Ilia Alshanetsky --- ## #25002 — Richard Quadling — 2006-07-27T12:52:44+00:00 URL: https://externals.io/message/24954#25002 · In reply to: #25000 I'm NOT a expert in this field. My POV may be quite dumb, but it may point to a solution. If memory is allocated, I would assume that somewhere there is a mechanism to access it otherwise this is a memory leak. So, rather than counting the memory in real time, why not have the `memory_get_usage()` (not sure about peak usage) simply iterate the used memory and calculate how much there is. This would put the overhead in the function call and not throughout the rest of the code. > > > > […] > > Counting memory all the time adds a fair amount of overhead to script > execution especially given how frequently PHP needs to allocate > memory. Therefore usage memory utilization functions are only > available when PHP is compiled with memory in which case it keeps > track of its mem usage. -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #25004 — bertrand Gugger — 2006-07-27T13:03:52+00:00 URL: https://externals.io/message/24954#25004 · In reply to: #25000 Ilia Alshanetsky wrote: > > > > > […] > > Counting memory all the time adds a fair amount of overhead to script > execution especially given how frequently PHP needs to allocate memory. > Therefore usage memory utilization functions are only available when > PHP is compiled with memory in which case it keeps track of its mem usage. That may hear off topic , but how you enable something else than 8M *without* getting this counting overhead ? It looks to me that --enable-memory-limit mixes 2 independant roles : - setting the amount of available memory, and enabling eventually local setting, - enabling the evaluation of the process consumption. I would like to be wrong on this :) -- toggg --- ## #25005 — Ilia Alshanetsky — 2006-07-27T13:07:44+00:00 URL: https://externals.io/message/24954#25005 · In reply to: #25004 > Ilia Alshanetsky wrote: > > […] > > That may hear off topic , but how you enable something else than 8M > *without* getting this counting overhead ? Eh? When you set a limit what are you trying to do? From my experience most of the time this setting is used by hosting providers to restrict memory utilization of various PHP scripts to avoid out-of- memory situations and system abuse. As such you'd want as accurate measurement as possible including any overhead and what not, since it is possible to use the overhead to exceed or in some cases even to bypass the memory limit entirely. > It looks to me that --enable-memory-limit mixes 2 independant roles : > - setting the amount of available memory, and enabling eventually > local setting, > - enabling the evaluation of the process consumption. That is correct. Memory limit by its nature requires PHP to count the amount of memory consumed, since such tracking is performed it enabled PHP to report internally via functions (or via Apache log) about its memory utilization. Ilia Alshanetsky --- ## #25017 — Ron Korving — 2006-07-27T17:17:35+00:00 URL: https://externals.io/message/24954#25017 · In reply to: #25005 Hi Ilia, "Ilia Alshanetsky" wrote in message news:70339B16-D50B-45B9-A55D-B58F9EEB679D@prohost.org... > Eh? When you set a limit what are you trying to do? From my > experience most of the time this setting is used by hosting providers > to restrict memory utilization of various PHP scripts to avoid out-of- > memory situations and system abuse. As such you'd want as accurate > measurement as possible including any overhead and what not, since it > is possible to use the overhead to exceed or in some cases even to > bypass the memory limit entirely. Yes, hosting providers would enable the memory limit. But who wants to use memory_get_[peak_]usage()? Not the hosting provider, but the application developer. - Ron --- ## #25018 — Brian Moon — 2006-07-27T17:23:14+00:00 URL: https://externals.io/message/24954#25018 · In reply to: #25017 > Yes, hosting providers would enable the memory limit. But who wants to use > memory_get_[peak_]usage()? Not the hosting provider, but the application > developer. We have some process, that if cache has gotten stale and needs to be recreated, the process can use 100MB or memory. We don't want to cap it. However, we do want to know about it and use apahce_child_terminate() to end that process. Right now, I have to actually read /proc to figure out my memory usage. I would much rather have a function. I guess I could enable a memory limit and set to like 1GB. Seems silly though. -- Brian Moon ------------- http://dealnews.com/ Its good to be cheap =) --- ## #25019 — Ilia Alshanetsky — 2006-07-27T17:33:27+00:00 URL: https://externals.io/message/24954#25019 · In reply to: #25017 > Yes, hosting providers would enable the memory limit. But who wants > to use > memory_get_[peak_]usage()? Not the hosting provider, but the > application > developer. > The peak usage function was added with profiling in mind and keeping track of scripts that spike memory, when I added it the intent was it would help with profiling and debugging. Not something used everyday in production environment. In dev environment speed is not an issue, so you can enable memory limit and raise the limit to some very high value such as 1Gb so it does not get in the way. Ilia Alshanetsky --- ## #25027 — bertrand Gugger — 2006-07-28T06:01:06+00:00 URL: https://externals.io/message/24954#25027 · In reply to: #25005 Ilia Alshanetsky wrote: > > > > > […] > > > Eh? When you set a limit what are you trying to do? From my experience > most of the time this setting is used by hosting providers to restrict > memory utilization of various PHP scripts to avoid out-of- memory > situations and system abuse. The most cases I saw were providers or production site wanting to grant *more* than 8M (typically 16M) to their customers / users. I dont call that a "restriction". You should know this 8M limit is reached very easily. e.g. lot of popular CMS wont work below this limit as soon as they have a litle contents. > As such you'd want as accurate measurement > as possible including any overhead and what not, since it is possible > to use the overhead to exceed or in some cases even to bypass the > memory limit entirely. You miss the point , they don't care about accuracy, they just don't want the default 8M but more. > > > […] > > > That is correct. Memory limit by its nature requires PHP to count the > amount of memory consumed, since such tracking is performed it enabled > PHP to report internally via functions (or via Apache log) about its > memory utilization. I have no idea how it works internally ... Are you meaning the default "hard-coded" 8M limit is magic and will not need to count the amount of memory consumed but any custom setting will ? -- toggg