# foreach() for strings

- Thread: https://externals.io/message/53383
- Original archive: https://news-web.php.net/php.internals/53383
- Started by Robert Eisele on 2011-06-20T11:27:06+00:00
- Messages: 60
- Page 1 of 3: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/53383.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #53383 — Robert Eisele — 2011-06-20T11:27:06+00:00

URL: https://externals.io/message/53383#53383

foreach() has many functions, looping over arrays, objects and implementing
the iterator interface. I think it's also quite intuitive to use foreach()
for strings, too.

If you want to implement a parser in PHP, you have to go the way with for +
strlen + `substr()` or $x[$i] to address one character of the string. We could
overdo the functionality of foreach()
by implementing LVAL's, too, in order to access single bits but this is
really uncommon, even if the way of thinking could be, that foreach() gives
a single attribute of each value, no matter
if it's a complex object with the iterator interface or a primitive. What do
you think about this one? My point of view is, that foreach() is very
useful, which was acknowledged by many ppl via the comments of my article.

I think, adding features like this persuades the one or the other PHP user
to upgrade to 5.4.

Robert

---

## #53388 — Derick Rethans — 2011-06-20T11:44:43+00:00

URL: https://externals.io/message/53383#53388 · In reply to: #53383

> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.

> If you want to implement a parser in PHP, you have to go the way with for +
> strlen + `substr()` or $x[$i] to address one character of the string.

Yes, this sounds like a good addition to me. One question though, what 
to do with an object that implements __toString() ?

> We could
> overdo the functionality of foreach()
> by implementing LVAL's, too, in order to access single bits but this is
> really uncommon, even if the way of thinking could be, that foreach() gives
> a single attribute of each value, no matter
> if it's a complex object with the iterator interface or a primitive. What do
> you think about this one? My point of view is, that foreach() is very
> useful, which was acknowledged by many ppl via the comments of my article.

I don't think we should do it for bits, as nothing in PHP really does do 
anything with that. If you want to do stuff with bits, I think the 
"bitset" package (http://pecl.php.net/package/Bitset) is the way 
forwards.

cheers,
Derick

-- 
http://derickrethans.nl | http://xdebug.org
Like Xdebug? Consider a donation: http://xdebug.org/donate.php
twitter: @derickr and @xdebug

---

## #53389 — Robert Eisele — 2011-06-20T11:48:08+00:00

URL: https://externals.io/message/53383#53389 · In reply to: #53388

2011/6/20 Derick Rethans <derick@php.net>

> 
>
> > […]
> implementing
> > […]
> foreach()
> > […]
>
> > […]
> +
> > […]
>
> Yes, this sounds like a good addition to me. One question though, what
> to do with an object that implements __toString() ?
>

That's the question, maybe one must force __toString() via an explicit
string-cast:

foreach( (string) $obj as $k=>$v)


>
> > […]
> gives
> > […]
> do
> > […]
> article.
>
> I don't think we should do it for bits, as nothing in PHP really does do
> anything with that. If you want to do stuff with bits, I think the
> "bitset" package (http://pecl.php.net/package/Bitset) is the way
> forwards.
>

yep, i totally agree.

>
> cheers,
> Derick
>
> --
> http://derickrethans.nl | http://xdebug.org
> Like Xdebug? Consider a donation: http://xdebug.org/donate.php
> twitter: @derickr and @xdebug

---

## #53390 — johannes@schlueters.de — 2011-06-20T11:56:37+00:00

URL: https://externals.io/message/53383#53390 · In reply to: #53383

> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.

I would prefer a TextIterator as we had in the old PHP 6 as this allows
more powerful filtering etc. using iterator semantics even though this
might be a bit slower.

johannes

---

## #53393 — Derick Rethans — 2011-06-20T12:06:57+00:00

URL: https://externals.io/message/53383#53393 · In reply to: #53390

> 
> > […]
> 
> I would prefer a TextIterator as we had in the old PHP 6 as this allows
> more powerful filtering etc. using iterator semantics even though this
> might be a bit slower.

I think TextIterator is another good addition, but it will make PHP 
depend on ICU. Therefore, I think just a foreach for strings seems 
valuable to me.

Derick

---

## #53394 — Derick Rethans — 2011-06-20T12:07:48+00:00

URL: https://externals.io/message/53383#53394 · In reply to: #53389

> 2011/6/20 Derick Rethans <derick@php.net>
> 
> > […]
> 
> That's the question, maybe one must force __toString() via an explicit 
> string-cast:
> 
> foreach( (string) $obj as $k=>$v)

That's a sensible thing indeed. We just need to make sure we document 
this with an example and a test then.

cheers,
Derick

-- 
http://derickrethans.nl | http://xdebug.org
Like Xdebug? Consider a donation: http://xdebug.org/donate.php
twitter: @derickr and @xdebug

---

## #53397 — Pierre Joye — 2011-06-20T12:09:23+00:00

URL: https://externals.io/message/53383#53397 · In reply to: #53390

2011/6/20 Johannes Schlüter <johannes@schlueters.de>:
> 
> > […]
>
> I would prefer a TextIterator as we had in the old PHP 6 as this allows
> more powerful filtering etc. using iterator semantics even though this
> might be a bit slower.

A foreach with string should be seen as binary buffer, with no clue
about its content and only to fetch it byte by byte.

TextIterator can be smarter and support unicode when ICU is available.

Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #53398 — Pierre Joye — 2011-06-20T12:16:08+00:00

URL: https://externals.io/message/53383#53398 · In reply to: #53383

hi Robert,

I would go with a RFC for that one, at least to document/cover edge
cases to help the doc team to properly document this change if it gets
approved.

Thanks for your work so far!


> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.
>
> If you want to implement a parser in PHP, you have to go the way with for +
> strlen + `substr()` or $x[$i] to address one character of the string. We could
> overdo the functionality of foreach()
> by implementing LVAL's, too, in order to access single bits but this is
> really uncommon, even if the way of thinking could be, that foreach() gives
> a single attribute of each value, no matter
> if it's a complex object with the iterator interface or a primitive. What do
> you think about this one? My point of view is, that foreach() is very
> useful, which was acknowledged by many ppl via the comments of my article.
>
> I think, adding features like this persuades the one or the other PHP user
> to upgrade to 5.4.
>
> Robert
>



-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #53401 — Lee davis — 2011-06-20T13:11:54+00:00

URL: https://externals.io/message/53383#53401 · In reply to: #53383

I think this would be quite a useful feature, and am In favor of it.
However, I think caution should be taken when shifting array utilities out
of their remit and allowing them to manipulate / traverse other data types.
You may see the floodgates opening for more request to adapt array functions
for other uses.

Say for instance..

Could we also use `current()`, `next()` and `key()` for iteration of strings?

$string = 'string';
while ($char = current($string))
{
echo key($string)   // Would output the offset position I assume 0,1,2 etc??
echo $char          // outputs each letter of string
next($string);
}

Lee



> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.
>
> If you want to implement a parser in PHP, you have to go the way with for +
> strlen + `substr()` or $x[$i] to address one character of the string. We
> could
> overdo the functionality of foreach()
> by implementing LVAL's, too, in order to access single bits but this is
> really uncommon, even if the way of thinking could be, that foreach() gives
> a single attribute of each value, no matter
> if it's a complex object with the iterator interface or a primitive. What
> do
> you think about this one? My point of view is, that foreach() is very
> useful, which was acknowledged by many ppl via the comments of my article.
>
> I think, adding features like this persuades the one or the other PHP user
> to upgrade to 5.4.
>
> Robert

---

## #53405 — Jonathan Bond-Caron — 2011-06-20T14:18:43+00:00

URL: https://externals.io/message/53383#53405 · In reply to: #53401

> 
> Could we also use `current()`, `next()` and `key()` for iteration of strings?
> 
> $string = 'string';
> while ($char = current($string))
> {
> echo key($string)   // Would output the offset position I assume 0,1,2
> etc??
> echo $char          // outputs each letter of string
> next($string);
> }
> 

Hopefully it can be supported without sacrificing too much performance
Like others mentioned, it seems important to distinguish between binary/byte
and text iteration.

$string = new ByteIterator('string é');
foreach($string as $i => $byte) 
  ...

$string = new TextIterator('string é');
foreach($string as $i => $char) 
  ...

When most developers get a 'string' from a database, my hunch is they assume
they would be iterating over the 'characters' (utf8, iso.. encoding) and not
individual bytes.

So +1 to string iteration as long as there's byte iteration and some plan
for text iteration / by character (with icu or not).

---

## #53413 — Ilia Alshanetsky — 2011-06-20T15:25:21+00:00

URL: https://externals.io/message/53383#53413 · In reply to: #53383

As long as it works on a premise that a "string" is a byte array and
each element represents 1 byte, +1 from me.


> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.
>
> If you want to implement a parser in PHP, you have to go the way with for +
> strlen + `substr()` or $x[$i] to address one character of the string. We could
> overdo the functionality of foreach()
> by implementing LVAL's, too, in order to access single bits but this is
> really uncommon, even if the way of thinking could be, that foreach() gives
> a single attribute of each value, no matter
> if it's a complex object with the iterator interface or a primitive. What do
> you think about this one? My point of view is, that foreach() is very
> useful, which was acknowledged by many ppl via the comments of my article.
>
> I think, adding features like this persuades the one or the other PHP user
> to upgrade to 5.4.
>
> Robert

---

## #53414 — John Crenshaw — 2011-06-20T15:25:56+00:00

URL: https://externals.io/message/53383#53414 · In reply to: #53401

> -----Original Message-----
> From: Lee davis [mailto:leedavis81@gmail.com] 
> Sent: Monday, June 20, 2011 9:12 AM
> To: Robert Eisele
> Cc: internals@lists.php.net
> Subject: Re: [PHP-DEV] foreach() for strings
> 
> I think this would be quite a useful feature, and am In favor of it.
> However, I think caution should be taken when shifting array utilities out
> of their remit and allowing them to manipulate / traverse other data types.
> You may see the floodgates opening for more request to adapt array functions
> for other uses.
> 
> Say for instance..
> 
> Could we also use `current()`, `next()` and `key()` for iteration of strings?
> 
> $string = 'string';
> while ($char = current($string))
> {
> echo key($string)   // Would output the offset position I assume 0,1,2 etc??
> echo $char          // outputs each letter of string
> next($string);
> }
> 
> Lee
> 
> 
> 
> > […]

Doing this with an explicit iterator object is a fine idea. The syntax becomes something like:

foreach(new TextIterator($s, 'UTF8') as $pos=>$c)
{
    ...
}

On the other hand, I think that trying to support iteration without using an iterator object to mediate would be a disaster, and I'm opposed to doing something like that because:
1. The code just looks wrong. PHP developers are generally insulated from the char-arrayness of strings. In addition, since PHP isn't typesafe, the code becomes highly ambiguous. Is the code iterating an array, or a string? It is very hard to tell just by looking. It may be convenient to write, but it's certainly not convenient to read or maintain later. On the other hand, with a mediating iterator object, the intent becomes obvious, and the code is highly readable.
2. The odds of iterating any given string are slim at best. Supporting current, key, next, etc. would require the string object internally to get bloated with additional unnecessary data that is almost never used. This bloat isn't a single int either. For optimal performance it would need to consist of no less than two size_t (char position and binary position), and one encoding indicator.
3. Iteration cannot work without knowing which encoding to use for the string. Is it UTF8? UTF16? UTF7? Binary or some single byte encoding? Some other exotic wide encoding? Without an iterator object in the middle, there is no way to specify this encoding. Always treating this as binary would also be a mistake, since this is almost certainly never actually the correct behavior, even though it may often appear to behave correctly with simple inputs.
4. I've had simple mistakes caught numerous times when foreach complains about getting a scalar rather than an array. So far, it has been exactly right every time. Allowing strings to be iterated would, in the name of convenience, increase the probability of stupid mistakes evading detection. Even worse, the code itself would look logically correct until the developer finally realizes that they have a string and not an array. Errors like this are probably far more common in most projects than the need to iterate a string, so making this change hurts debugging in the common case, for the sake of syntactic sugar in the rare case. Not a good trade.

John Crenshaw
Priacta, Inc.

---

## #53424 — John Crenshaw — 2011-06-20T16:15:36+00:00

URL: https://externals.io/message/53383#53424 · In reply to: #53413

> From: Ilia Alshanetsky [mailto:ilia@prohost.org] 
> 
> As long as it works on a premise that a "string" is a byte array and
> each element represents 1 byte, +1 from me.

Code written on this premise is almost always bug central when people finally get around to realizing why they really do need to support wide characters (and everybody does, because people like to paste stuff containing non-break-spaces, and decorative quotes). I really don't think this single byte character mentality should be encouraged.

Also, how do you think this will work with the Unicode conversion in PHP 6? Guaranteed, this will break stuff. Some people will have written code to iterate characters, assuming single byte characters, some people will have written code ACTUALLY intending to iterate as a byte array. Sadly, we can almost certainly assume that the single byte characters assumption (which is wrong) will also be, by far, the most common. Supporting that most common case when moving to PHP 6 would require breaking the binary case (which was the only properly written code in the first place.) On the other hand, supporting the binary case means breaking the most common case.

John Crenshaw
Priacta, Inc.

---

## #53423 — Stas Malyshev — 2011-06-20T16:31:43+00:00

URL: https://externals.io/message/53383#53423 · In reply to: #53383

Hi!

> foreach() has many functions, looping over arrays, objects and implementing
> the iterator interface. I think it's also quite intuitive to use foreach()
> for strings, too.

I'm not sure how you'd implement such thing, but then I think things 
like `next()`, `end()`, etc. should work too...
-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #53428 — Stas Malyshev — 2011-06-20T16:49:22+00:00

URL: https://externals.io/message/53383#53428 · In reply to: #53424

Hi!


> > […]
>
> Code written on this premise is almost always bug central when people
> finally get around to realizing why they really do need to support
> wide characters (and everybody does, because people like to paste
> stuff containing non-break-spaces, and decorative quotes). I really
> don't think this single byte character mentality should be
> encouraged.

I think you're right, TextIterator would be better (and also much easier 
to implement, I think). Didn't we have it in Unicode branch? We could 
port it back or we could have something along the lines of 
grapheme_extract...

> Also, how do you think this will work with the Unicode conversion in
> PHP 6? Guaranteed, this will break stuff. Some people will have

I don't think we need to worry about PHP 6 now... If we ever get back to 
Unicode support, it probably will be different.
-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

---

## #53470 — Derick Rethans — 2011-06-21T10:53:56+00:00

URL: https://externals.io/message/53383#53470 · In reply to: #53428

> 
> > […]
> 
> I think you're right, TextIterator would be better (and also much easier to
> implement, I think). Didn't we have it in Unicode branch? We could port it
> back or we could have something along the lines of grapheme_extract...

It depended on ICU there, and I would be against making a core thing in 
PHP 5.x depend on ICU.

cheers,
Derick

-- 
http://derickrethans.nl | http://xdebug.org
Like Xdebug? Consider a donation: http://xdebug.org/donate.php
twitter: @derickr and @xdebug

---

## #53472 — Pierre Joye — 2011-06-21T11:03:23+00:00

URL: https://externals.io/message/53383#53472 · In reply to: #53470

> It depended on ICU there, and I would be against making a core thing in
> PHP 5.x depend on ICU.

It can and should be done as part of intl, actually.

But that's somehow unrelated to the proposal here, as it is about
byte, not characters :)

-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #53477 — Lester Caine — 2011-06-21T11:33:32+00:00

URL: https://externals.io/message/53383#53477 · In reply to: #53472

Pierre Joye wrote:
> > […]
> It can and should be done as part of intl, actually.
>
> But that's somehow unrelated to the proposal here, as it is about
> byte, not characters :)

I believe this may be where some of the new niggles may be coming from? With 
browsers returning unicode, it may be that some of the 'extra' characters are 
being returned as multibyte rather than as single bytes? Such as the problem 
reported on the general list currently. How do we ensure that we are dealing 
with single byte character strings nowadays?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk//
Firebird - http://www.firebirdsql.org/index.php

---

## #53478 — Pierre Joye — 2011-06-21T11:42:45+00:00

URL: https://externals.io/message/53383#53478 · In reply to: #53477

> Pierre Joye wrote:
> > […]
>
> I believe this may be where some of the new niggles may be coming from? With
> browsers returning unicode, it may be that some of the 'extra' characters
> are being returned as multibyte rather than as single bytes? Such as the
> problem reported on the general list currently. How do we ensure that we are
> dealing with single byte character strings nowadays?

As it has been stated numerous times in this thread and other, we do
not do anything with multi bytes systems, unicode, etc. mbstring and
intl do, but php's string as of now is all about bytes, array of bytes
if I may describe them this way.

And we can't change this behavior.

Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #53479 — Lester Caine — 2011-06-21T12:29:23+00:00

URL: https://externals.io/message/53383#53479 · In reply to: #53478

Pierre Joye wrote:
> 
> > […]
>
> As it has been stated numerous times in this thread and other, we do
> not do anything with multi bytes systems, unicode, etc. mbstring and
> intl do, but php's string as of now is all about bytes, array of bytes
> if I may describe them this way.
>
> And we can't change this behavior.

That is exactly the point. I suppose what I am asking is how people ensure that 
what they are feeding into simple strings are single byte when cut and past 
nowadays does not make a distinction?

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk//
Firebird - http://www.firebirdsql.org/index.php

---

## #53480 — John Crenshaw — 2011-06-21T14:38:22+00:00

URL: https://externals.io/message/53383#53480 · In reply to: #53479

Pierre Joye wrote:
> 
> > […]
>
> As it has been stated numerous times in this thread and other, we do
> not do anything with multi bytes systems, unicode, etc. mbstring and
> intl do, but php's string as of now is all about bytes, array of bytes
> if I may describe them this way.
>
> And we can't change this behavior.

This mindset is fundamentally broken. You can call it a byte array all you want, but the truth is that 99.999% of the time, when a developer is using a string they need it for characters, not for bytes, and characters are not single byte. Even English users tend to submit Unicode range characters at an alarming rate. If you're using a WYSIWYG editor, Chrome will submit non-breaking-spaces as the actual UTF8 encoded character, not as an HTML encoded entity. Whether developers like it, or even know it, supporting an extended universal character set is not really optional.

PHP makes this bad enough with the whole collection of bytewise string functions, including many with no appropriate multibyte aware replacement, but at least this can be avoided, quickly audited, and in the future can even be fixed in any number of ways with only a nominal BC impact. Hard coding this single byte idiocy into a language construct (foreach) though would be an incredibly awful idea. This would create a trap for new naive PHP developers, and create a character set problem that the language could NEVER recover from without a massive BC break.

This proposal is really about adding a feature which whenever it used is almost guaranteed to be an error. It probably won't look to the developer like an error during simple testing, but will almost certainly show up as an error in production. Is it really worth all that for a bit of syntax sugar that the developer will have to strip out anyway to fix their bug?

If string iteration needs to be addressed in the core (and IMO it doesn't because it can be handled at the script level, but if it does) why not use iterator classes? This gives the same functionality and prevents the language from encouraging hidden bugs.

John Crenshaw
Priacta, Inc.

---

## #53481 — Pierre Joye — 2011-06-21T15:13:19+00:00

URL: https://externals.io/message/53383#53481 · In reply to: #53480

> This mindset is fundamentally broken. You can call it a byte array all you want, but the truth is that 99.999% of the time, when a developer is using a string they need it for characters, not for bytes

Let me rephrase:

For backward compatibility reasons we cannot change this behavior.

Any serious text processing should be done using intl, mbstring,
transliterator (pecl) or other similar solutions.

Cheers,
--
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

---

## #53483 — Tomas Kuliavas — 2011-06-21T15:55:39+00:00

URL: https://externals.io/message/53383#53483 · In reply to: #53480

2011.06.21 17:38 John Crenshaw rašė:
> Pierre Joye wrote:
> > […]
>
> This mindset is fundamentally broken. You can call it a byte array all you
> want, but the truth is that 99.999% of the time, when a developer is using
> a string they need it for characters, not for bytes, and characters are
> not single byte. Even English users tend to submit Unicode range
> characters at an alarming rate. If you're using a WYSIWYG editor, Chrome
> will submit non-breaking-spaces as the actual UTF8 encoded character, not
> as an HTML encoded entity. Whether developers like it, or even know it,
> supporting an extended universal character set is not really optional.

They submit it in utf-8 only if your html form allows them to do that or
they don't follow html specification and try to exploit your form. Set
form input charset to iso-8859-1 and your nbspace will take only one byte.

-- 
Tomas

---

## #53485 — Arvids Godjuks — 2011-06-21T16:05:59+00:00

URL: https://externals.io/message/53383#53485 · In reply to: #53483

As a userland developer due to my geographical nature i have to work with 3
languages constantly - english, russian (cyryllic) and latvian (witch has
it's own share of non latin characters). I end up using utf-8 in every
project. And some give me a headake of dealing with text parsing. mb_string
covers just part of the functionality and can be turned off.

I personally think something has to be done about unicode handling in php
after 5.4 so that we have an official method of dealing with it in the core.
Probably it can be done in a namespace of its own and be new functionality
to witch people should migrate.

my 2 cents.
21.06.2011 17:56 пользователь "Tomas Kuliavas" <tokul@users.sourceforge.net>
написал:
> 2011.06.21 17:38 John Crenshaw rašė:
> > […]
you
> > […]
using
> > […]
>
> They submit it in utf-8 only if your html form allows them to do that or
> they don't follow html specification and try to exploit your form. Set
> form input charset to iso-8859-1 and your nbspace will take only one byte.
>
> --
> Tomas

---

## #53486 — Reindl Harald — 2011-06-21T16:14:40+00:00

URL: https://externals.io/message/53383#53486 · In reply to: #53483

Am 21.06.2011 17:55, schrieb Tomas Kuliavas:

> They submit it in utf-8 only if your html form allows them to do that or
> they don't follow html specification and try to exploit your form. Set
> form input charset to iso-8859-1 and your nbspace will take only one byte.

and this naive attitude is the root of most security problems!

why do you believe that every client submission is coming over
your form or generally over anything you can control?

---

Next page: https://externals.io/message/53383.md?page=2