Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:34994
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 204.11.219.139 cause and error)
Message-ID: <479EB7ED.8070606@lerdorf.com>
Date: Mon, 28 Jan 2008 21:21:49 -0800
User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031)
MIME-Version: 1.0
To: Peter Brodersen <penguin@php.net>
CC: internals Mailing List <internals@lists.php.net>
References: <200801241426.39756.arnaud.lb@gmail.com> <479A613C.8030604@zend.com> <3hpsp3hmn2de4fard8lkpentg24k70jrhg@4ax.com> <479E80D8.5020206@lerdorf.com> <ci4tp3hro32ukdru6ik4d3t0kqsfounccf@4ax.com>
In-Reply-To: <ci4tp3hro32ukdru6ik4d3t0kqsfounccf@4ax.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] [PATCH] Bug #43896 htmlspecialchars returns empty stringoninvalid
 unicode sequence
From: rasmus@lerdorf.com (Rasmus Lerdorf)

Peter Brodersen wrote:
> http://php.net/xml also documents this replacement:
> ==
> If PHP encounters characters in the parsed XML document that can not be
> represented in the chosen target encoding, the problem characters will be
> "demoted". Currently, this means that such characters are replaced by a
> question mark.
> ==

That was back in the expat days.  We don't use that xml parser anymore.

> http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt mentions:
> ==
> According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
> receiving UTF-8 shall interpret a "malformed sequence in the same way
> that it interprets a character that is outside the adopted subset" and
> "characters that are not within the adopted subset shall be indicated
> to the user" by a receiving device. A quite commonly used approach in
> UTF-8 decoders is to replace any malformed UTF-8 sequence by a
> replacement character (U+FFFD), which looks a bit like an inverted
> question mark, or a similar symbol. It might be a good idea to
> visually distinguish a malformed UTF-8 sequence from a correctly
> encoded Unicode character that is just not available in the current
> font but otherwise fully legal, even though ISO 10646-1 doesn't
> mandate this. In any case, just ignoring malformed sequences or
> unavailable characters does not conform to ISO 10646, will make
> debugging more difficult, and can lead to user confusion.
> ==

That part is completely different.  That's at the display level.
Replacing it in the backend makes no sense to me.  Don't use
utf8_decode.  Use iconv() so you know what the heck is going on.

-Rasmus