Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:111984
To: Thomas Landauer <thomas@landauer.at>, internals@lists.php.net
References: <cd1a914c-aeb2-9c73-147c-bb49a78318a5@landauer.at>
Message-ID: <2c54d906-35c7-6ed6-ac47-649b7bcde2a2@gmx.de>
Date: Fri, 2 Oct 2020 14:24:36 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.3.1
MIME-Version: 1.0
In-Reply-To: <cd1a914c-aeb2-9c73-147c-bb49a78318a5@landauer.at>
Content-Type: text/plain; charset=utf-8
Content-Language: de-DE
Content-Transfer-Encoding: quoted-printable
Subject: Re: Suggestion: Make all PCRE functions return *character*
 offsets,rather than *byte* offsets if the modifier `u` (PCRE_UTF8) is given
From: cmbecker69@gmx.de ("Christoph M. Becker")

On 10/2/2020 at 12:01 PM, Thomas Landauer wrote:

> this is a follow-up of a bug I opened, and cmb suggested to continue
> here: https://bugs.php.net/bug.php?id=3D80166

Indeed, thanks!

> Advantages:
>
> 1: Easier string manipulation:
> If somebody does (as in my case) `preg_match_all()` with
> PREG_OFFSET_CAPTURE, what will they probably use those returned
> numbers/offsets for?
> My answer: For *splitting the string* - in some way or the other. Now,
> with byte offsets, I can't do such basic things as just `+1` to get to
> the next character. Or extract exactly 3 characters.

The term "character" is ambiguous wrt. Unicode.  The mbstring functions
work on Unicode code points, so it's probably better to use that term
instead.

While it is trivial to get the next code point using index+1, this is
not necessarily the next character, as perceived by a human.  Using
mb_substr(), you may even break "characters", e.g. <https://3v4l.org/5geOr=
>.

> 2: Better performance:
> This may sound odd, since cmb said the exact opposite ;-) (sequential
> access vs. random access). However, if I need character offsets (see 1),
> what can I do? I'm forced to use some workaround on top - as e.g.
> https://www.php.net/manual/en/function.preg-match-all.php#71572 - which
> is certainly way slower than any native implementation.

If mbstring functions are used to find some offset, they always have to
traverse the string from the beginning, even if you are just interested
in the last code point of a long string.  If you have byte offsets, that
code point can be accessed directly.  Of course, that may not suit any
possible scenario, but I still don't think that the PCRE functions
should deal with code point offset instead of byte offsets.

Regards,
Christoph