# default charset confusion
- Thread: https://externals.io/message/58853
- Original archive: https://news-web.php.net/php.internals/58853
- Started by Rasmus Lerdorf on 2012-03-12T06:49:27+00:00
- Messages: 39
- Page 1 of 2: messages 1 to 25 (25 per page, add `?page=N` to the URL)
- Next page: https://externals.io/message/58853.md?page=2
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt
---
## #58853 — Rasmus Lerdorf — 2012-03-12T06:49:27+00:00
URL: https://externals.io/message/58853#58853
I caused this situation myself by not explicitly differentiating between
the default charset for the internal `htmlspecialchars()` and
`htmlentities()` functions and the output charset directive ini directive
default_charset.
The idea behind the default_charset ini directive was to act as the
charset that gets specified in the HTTP Content-type header if you do
not explicitly send your own Content-type header with the `header()`
function. This has been muddied a bit by the fact that
htmlspecialchars/htmlentities can take it into account when it is trying
to choose which encoding to use when handling data passed to it. This
isn't done by default since it actually makes little sense. It is only
done if you pass an empty string as the encoding argument. If you don't
pass anything at all the default is UTF-8 in 5.4. In 5.3 this was
ISO-8859-1.
And here is where the confusion comes in. We, myself included, have told
people that they can get the 5.3 behaviour back by setting the
default_charset ini directive to iso-8859-1. But, this is only true if
they are forcing htmlspecialchars/htmlentities to check that setting
with an empty string as the encoding arg. Most apps just do
htmlspecialchars($str) and nothing else. Plus, it is really not a good
idea to tie the internal encoding of data being passed to these
functions to the output charset. You should be able to change the output
charset without worrying about your runtime encoding at that level.
What this effectively means is that we are asking people to go through
their code and add an explicit charset to all `htmlspecialchars()` and
`htmlentities()` calls. I think this will be a hurdle for 5.4 adoption.
What we really need is what we added in PHP 6. A runtime encoding ini
setting that is distinct from the output charset which we can use here.
That would allow people to fix all their legacy code to a specific
runtime encoding with a single ini setting instead of changing thousands
of lines of code. I propose that we add such a directive to 5.4.1 to
ease migration.
See https://bugs.php.net/61354 for the first signs of grumbling about
this one. As more people migrate I have a feeling this will end up being
the most difficult part of the migration.
-Rasmus
---
## #58854 — Laruence — 2012-03-12T06:53:17+00:00
URL: https://externals.io/message/58853#58854 · In reply to: #58853
> I caused this situation myself by not explicitly differentiating between
> the default charset for the internal `htmlspecialchars()` and
> `htmlentities()` functions and the output charset directive ini directive
> default_charset.
>
> The idea behind the default_charset ini directive was to act as the
> charset that gets specified in the HTTP Content-type header if you do
> not explicitly send your own Content-type header with the `header()`
> function. This has been muddied a bit by the fact that
> htmlspecialchars/htmlentities can take it into account when it is trying
> to choose which encoding to use when handling data passed to it. This
> isn't done by default since it actually makes little sense. It is only
> done if you pass an empty string as the encoding argument. If you don't
> pass anything at all the default is UTF-8 in 5.4. In 5.3 this was
> ISO-8859-1.
>
> And here is where the confusion comes in. We, myself included, have told
> people that they can get the 5.3 behaviour back by setting the
> default_charset ini directive to iso-8859-1. But, this is only true if
> they are forcing htmlspecialchars/htmlentities to check that setting
> with an empty string as the encoding arg. Most apps just do
> htmlspecialchars($str) and nothing else. Plus, it is really not a good
> idea to tie the internal encoding of data being passed to these
> functions to the output charset. You should be able to change the output
> charset without worrying about your runtime encoding at that level.
>
> What this effectively means is that we are asking people to go through
> their code and add an explicit charset to all `htmlspecialchars()` and
> `htmlentities()` calls. I think this will be a hurdle for 5.4 adoption.
>
> What we really need is what we added in PHP 6. A runtime encoding ini
> setting that is distinct from the output charset which we can use here.
> That would allow people to fix all their legacy code to a specific
> runtime encoding with a single ini setting instead of changing thousands
> of lines of code. I propose that we add such a directive to 5.4.1 to
> ease migration.
+1, especially for non-utf8 applications.
thanks
>
> See https://bugs.php.net/61354 for the first signs of grumbling about
> this one. As more people migrate I have a feeling this will end up being
> the most difficult part of the migration.
>
> -Rasmus
>
> --
>
>
>
--
Laruence Xinchen Hui
http://www.laruence.com/
---
## #58855 — Adam Jon Richardson — 2012-03-12T07:00:42+00:00
URL: https://externals.io/message/58853#58855 · In reply to: #58853
> What we really need is what we added in PHP 6. A runtime encoding ini
> setting that is distinct from the output charset which we can use here.
> That would allow people to fix all their legacy code to a specific
> runtime encoding with a single ini setting instead of changing thousands
> of lines of code. I propose that we add such a directive to 5.4.1 to
> ease migration.
>
This seems likes a very reasonable way of dealing with this issue.
Adam
---
## #58856 — Stas Malyshev — 2012-03-12T07:10:10+00:00
URL: https://externals.io/message/58853#58856 · In reply to: #58853
Hi!
> What we really need is what we added in PHP 6. A runtime encoding ini
> setting that is distinct from the output charset which we can use here.
> That would allow people to fix all their legacy code to a specific
> runtime encoding with a single ini setting instead of changing thousands
> of lines of code. I propose that we add such a directive to 5.4.1 to
> ease migration.
One more charset INI setting? I'm not sure I like this. We have tons of
INIs already, and adding a new one each time we change something makes
both writing applications and configuring servers harder.
But as the manual says, ISO-8859-1 and UTF-8 are the same for
`htmlspecialchars()` - is it wrong? If yes, what exactly is the different
between old and new behavior? I tried to read #61354 but could make
little sense out of it, it lacks expected result and I have hard time
understanding what is the problem there. Could you explain?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #58857 — Laruence — 2012-03-12T07:32:09+00:00
URL: https://externals.io/message/58853#58857 · In reply to: #58856
> Hi!
>
>
> > […]
>
>
> One more charset INI setting? I'm not sure I like this. We have tons of INIs
> already, and adding a new one each time we change something makes both
> writing applications and configuring servers harder.
> But as the manual says, ISO-8859-1 and UTF-8 are the same for
> `htmlspecialchars()` - is it wrong? If yes, what exactly is the different
> between old and new behavior? I tried to read #61354 but could make little
> sense out of it, it lacks expected result and I have hard time understanding
> what is the problem there. Could you explain?
Hi:
if the argument string passed to htmlspecialchars is not in the
charset the htmlspecialchars expected(default is UTF8, and there is
only one way out is specific the third argument),
a empty string will returned without any notice or warning ;)
thanks
>
> --
> Stanislav Malyshev, Software Architect
> SugarCRM: http://www.sugarcrm.com/
> (408)454-6900 ext. 227
>
>
> --
>
>
>
--
Laruence Xinchen Hui
http://www.laruence.com/
---
## #58858 — Laruence — 2012-03-12T07:36:04+00:00
URL: https://externals.io/message/58853#58858 · In reply to: #58856
> Hi!
>
>
> > […]
>
>
> One more charset INI setting? I'm not sure I like this. We have tons of INIs
If we will definitely add a run_time_charset in the furture, then I
think it's okey add it now. :)
thanks
> already, and adding a new one each time we change something makes both
> writing applications and configuring servers harder.
> But as the manual says, ISO-8859-1 and UTF-8 are the same for
> `htmlspecialchars()` - is it wrong? If yes, what exactly is the different
> between old and new behavior? I tried to read #61354 but could make little
> sense out of it, it lacks expected result and I have hard time understanding
> what is the problem there. Could you explain?
>
> --
> Stanislav Malyshev, Software Architect
> SugarCRM: http://www.sugarcrm.com/
> (408)454-6900 ext. 227
>
>
> --
>
>
>
--
Laruence Xinchen Hui
http://www.laruence.com/
---
## #58859 — Rasmus Lerdorf — 2012-03-12T07:41:08+00:00
URL: https://externals.io/message/58853#58859 · In reply to: #58856
> Hi!
>
> > […]
>
> One more charset INI setting? I'm not sure I like this. We have tons of
> INIs already, and adding a new one each time we change something makes
> both writing applications and configuring servers harder.
> But as the manual says, ISO-8859-1 and UTF-8 are the same for
> `htmlspecialchars()` - is it wrong? If yes, what exactly is the different
> between old and new behavior? I tried to read #61354 but could make
> little sense out of it, it lacks expected result and I have hard time
> understanding what is the problem there. Could you explain?
Yes, it is a bit hard to understand from the bug report because
bugs.php.net is all utf-8, but we are talking about non utf-8 apps here.
This script should illustrate it: ( https://gist.github.com/2020502 )
$gb2312 = iconv('UTF-8','GB2312','我是测试');
$string = $string = "$gb2312
";
echo htmlspecialchars($string);
If you run that in PHP 5.3 you get:
<pre><p>���Dz���</p></pre>
The garbage-like chars there - if you don't see them, see
https://gist.github.com/2020442 - is the expected output. In PHP 5.4 the
output is nothing. The function recognizes that this is not valid UTF-8
and dumps the entire string.
Ignoring 5.4 for a second, if you in 5.3 do this:
echo htmlspecialchars($string);
echo htmlspecialchars($string, NULL, "ISO-8859-1");
echo htmlspecialchars($string, NULL, "UTF-8");
You will see that the first two output the escaped string with the
GB2312 bytes intact within it and the UTF-8 calls returns false because
it correctly recognizes that GB2312 is not UTF-8. We don't have any such
check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
`htmlspecialchars()` is wrong for PHP 5.3 as well as for 5.4.
And as expected, under 5.4 because the default is now the UTF-8
behaviour only the second echo gives a result.
-Rasmus
---
## #58860 — Rasmus Lerdorf — 2012-03-12T07:43:11+00:00
URL: https://externals.io/message/58853#58860 · In reply to: #58859
> $string = $string = "$gb2312
";
Sorry typo there obviously. Just one $string
-Rasmus
---
## #58861 — Stas Malyshev — 2012-03-12T07:52:41+00:00
URL: https://externals.io/message/58853#58861 · In reply to: #58859
Hi!
> Ignoring 5.4 for a second, if you in 5.3 do this:
>
> echo htmlspecialchars($string);
> echo htmlspecialchars($string, NULL, "ISO-8859-1");
> echo htmlspecialchars($string, NULL, "UTF-8");
>
> You will see that the first two output the escaped string with the
> GB2312 bytes intact within it and the UTF-8 calls returns false because
> it correctly recognizes that GB2312 is not UTF-8. We don't have any such
> check for 8859-1, so yes, saying UTF-8 and 8859-1 are the same for
> `htmlspecialchars()` is wrong for PHP 5.3 as well as for 5.4.
So the difference is that ISO8859-1 does not validate but UTF-8 validates?
I'm not sure what GB2312 encoding does but isn't it dangerous to do
`htmlspecialchars()` with wrong encoding? Wouldn't `htmlentities()` also
produce wrong result when used with wrong encoding?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #58862 — Adam Jon Richardson — 2012-03-12T08:09:34+00:00
URL: https://externals.io/message/58853#58862 · In reply to: #58861
On Mon, Mar 12, 2012 at 3:52 AM, Stas Malyshev wrote:
> Hi!
>
>
> Ignoring 5.4 for a second, if you in 5.3 do this:
> > […]
>
> So the difference is that ISO8859-1 does not validate but UTF-8 validates?
> I'm not sure what GB2312 encoding does but isn't it dangerous to do
> `htmlspecialchars()` with wrong encoding? Wouldn't `htmlentities()` also
> produce wrong result when used with wrong encoding?
The EUC-CN encoding appears to ensure compatibility with ascii by avoiding
the ascii range for each of its two bytes, so it seems that
htmlspecialchars should work OK:
http://en.wikipedia.org/wiki/GB_2312#EUC-CN
http://php.net/manual/en/mbstring.supported-encodings.php
Adam
Adam
---
## #58863 — Rasmus Lerdorf — 2012-03-12T08:11:58+00:00
URL: https://externals.io/message/58853#58863 · In reply to: #58861
> Hi!
>
> > […]
>
> So the difference is that ISO8859-1 does not validate but UTF-8 validates?
> I'm not sure what GB2312 encoding does but isn't it dangerous to do
> `htmlspecialchars()` with wrong encoding? Wouldn't `htmlentities()` also
> produce wrong result when used with wrong encoding?
Not sure you can validate 8859-1 since it isn't multibyte, can you? Is
there any byte that is explicitly forbidden in 8859-1?
And yes, it may very well be dangerous to use the wrong charset and now
that we have better support for GB2312 and other asian charsets in the
entities functions in 5.4 it is even more prudent to choose the right
one so we should provide some way to help people get it right short of
changing every call.
Gustavo suggested we could use the multibyte encoding setting.
Unfortunately only zend.script_encoding is available and I think
internal_encoding is closer to what we need here, but that is only
available as mbstring.internal_encoding.
-Rasmus
---
## #58865 — Yasuo Ohgaki — 2012-03-12T10:05:03+00:00
URL: https://externals.io/message/58853#58865 · In reply to: #58863
Hi
I think following PHP 5.4.0 NEWS entry is misleading.
. Changed default value of "default_charset" php.ini option from ISO-8859-1 to
UTF-8. (Rasmus)
I thought default_charset became UTF-8, so I was expecting
following HTTP header.
content-type text/html; charset=UTF-8
However, I got empty charset (missing 'charset=UTF-8').
So I looked up to source and found the line in SAPI.h
293 #define SAPI_DEFAULT_CHARSET ""
Empty string should be "UTF-8", isn't it?
BTW, empty charset in HTTP header does not mean the default will
be ISO-8859-1, but it let browser guess the encoding is used.
Guessing encoding may cause XSS under certain conditions.
Anyway, I was curious so I've checked ext/standard/html.c and found
/* {{{ entity_charset determine_charset
* returns the charset identifier based on current locale or a hint.
* defaults to UTF-8 */
static enum entity_charset determine_charset(char *charset_hint TSRMLS_DC)
{
int i;
enum entity_charset charset = cs_utf_8;
int len = 0;
const zend_encoding *zenc;
/* Default is now UTF-8 */
if (charset_hint == NULL)
return cs_utf_8;
There are 2 problems.
- php.ini's default_charset should be UTF-8.
- determine_charset() should not blindly default to UTF-8 when there
are no hint.
Old htmlentities/htmlspecialchars actually determines charset from
default_charset/mbstring.internal_encoding/etc. I think old behavior
is better than now.
How about make determine_charset() behaves like 5.3 and set the
SAPI_DEFAULT_CHARSET to "UTF-8"?
Then PHP will behave like as NEWS mentions, htmlentities/htmlspecialchars
default encoding became 'UTF-8' and users will have control for default
htmlenties/htmlspecialchars encoding.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #58866 — Yasuo Ohgaki — 2012-03-12T10:21:20+00:00
URL: https://externals.io/message/58853#58866 · In reply to: #58865
Hi,
I think motivation of
/* Default is now UTF-8 */
if (charset_hint == NULL)
return cs_utf_8;
is for better performance and I think it's good for better performance.
Alternative of my suggestion is introduce new php.ini entry as Rusmus
mentioned.
The name may be "default_html_escape_encoding"?
We should document this behavior very well, since it affects all of
non UTF-8 web sites.
Regards,
--
Yasuo Ohgaki
yohgaki@ohgaki.net
---
## #58872 — Laruence — 2012-03-12T12:48:22+00:00
URL: https://externals.io/message/58853#58872 · In reply to: #58866
> Hi,
>
> I think motivation of
>
> /* Default is now UTF-8 */
> if (charset_hint == NULL)
> return cs_utf_8;
>
> is for better performance and I think it's good for better performance.
> Alternative of my suggestion is introduce new php.ini entry as Rusmus
> mentioned.
>
> The name may be "default_html_escape_encoding"?
Hi:
in consideration of succinctness, I think run_time_encoding is better.
and we should also separate the determine_output_charset and
determine_run_time_charset(there is only one determin_charset now)
thanks
>
> We should document this behavior very well, since it affects all of
> non UTF-8 web sites.
>
> Regards,
>
> --
> Yasuo Ohgaki
> yohgaki@ohgaki.net
>
> --
>
>
>
--
Laruence Xinchen Hui
http://www.laruence.com/
---
## #58879 — Rasmus Lerdorf — 2012-03-12T15:27:18+00:00
URL: https://externals.io/message/58853#58879 · In reply to: #58865
> Hi
>
> I think following PHP 5.4.0 NEWS entry is misleading.
>
> . Changed default value of "default_charset" php.ini option from ISO-8859-1 to
> UTF-8. (Rasmus)
Yes, I have fixed that now.
> I thought default_charset became UTF-8, so I was expecting
> following HTTP header.
>
> content-type text/html; charset=UTF-8
>
> However, I got empty charset (missing 'charset=UTF-8').
> So I looked up to source and found the line in SAPI.h
>
> 293 #define SAPI_DEFAULT_CHARSET ""
>
> Empty string should be "UTF-8", isn't it?
No, we can't force an output charset on people since it would end up
breaking a lot of sites.
> - php.ini's default_charset should be UTF-8.
> - determine_charset() should not blindly default to UTF-8 when there
> are no hint.
>
> Old htmlentities/htmlspecialchars actually determines charset from
> default_charset/mbstring.internal_encoding/etc. I think old behavior
> is better than now.
>
> How about make determine_charset() behaves like 5.3 and set the
> SAPI_DEFAULT_CHARSET to "UTF-8"?
PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
if (charset_hint == NULL)
return cs_8859_1;
and in 5.4 we have:
if (charset_hint == NULL)
return cs_utf_8;
So there is no difference in their guessing when there is no hint, the
only difference is that in 5.4 we choose utf8 and in 5.3 we choose
8859-1 in that case.
-Rasmus
---
## #58880 — Michael Stowe — 2012-03-12T15:39:54+00:00
URL: https://externals.io/message/58853#58880 · In reply to: #58879
I think the ini directive, while adding another to the list, may be the
most unobtrusive method to address this issue, at least for developers.
I definitely agree with Rasmus that this could be one of the bigger
headaches in transitioning to 5.4 (for non-UTF8 sites) and unless we can
come up with a better solution, I say let's move forward with it for 5.4.1.
- Mike
>
> > […]
> ISO-8859-1 to
> > […]
>
> Yes, I have fixed that now.
>
> > […]
>
> No, we can't force an output charset on people since it would end up
> breaking a lot of sites.
>
> > […]
>
> PHP 5.3's determine_charset behaves exactly like 5.4's. In 5.3 we have:
>
> if (charset_hint == NULL)
> return cs_8859_1;
>
> and in 5.4 we have:
>
> if (charset_hint == NULL)
> return cs_utf_8;
>
> So there is no difference in their guessing when there is no hint, the
> only difference is that in 5.4 we choose utf8 and in 5.3 we choose
> 8859-1 in that case.
>
> -Rasmus
>
> --
>
>
>
>
--
-----------------------
"My command is this: Love each other as I
have loved you." John 15:12
---
## #58882 — Richard Lynch — 2012-03-12T19:31:32+00:00
URL: https://externals.io/message/58853#58882 · In reply to: #58853
> What we really need is what we added in PHP 6. A runtime encoding ini
> setting that is distinct from the output charset which we can use
> here.
The usual argument against another php.ini setting, other than "too
many already" is the difficulty it presents to write portable code
libraries.
I'm not smart enough to predict how such a setting (regardless of its
name) would help or hinder a library of code that doesn't want another
conditional in a zillion places.
But you folks are that smart. :-)
And I haven't seen any discussion regarding this sub-issue.
So, how would the help / hinder authors of generic library code to be
distributed in the wild?
Forgive me if the answer is so blindingly obvious I should already
know it... :-)
--
brain cancer update:
http://richardlynch.blogspot.com/search/label/brain%20tumor
Donate:
https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=FS9NLTNEEKWBE
---
## #58884 — Stas Malyshev — 2012-03-12T19:40:56+00:00
URL: https://externals.io/message/58853#58884 · In reply to: #58863
Hi!
> And yes, it may very well be dangerous to use the wrong charset and now
> that we have better support for GB2312 and other asian charsets in the
> entities functions in 5.4 it is even more prudent to choose the right
> one so we should provide some way to help people get it right short of
> changing every call.
I'm not sure "changing every call" is such a big problem - it's one grep
and one replace, can be done in one line of sed/awk/perl/php probably.
But a bigger issue is here that people insist on using wrong charsets
and expect language to have some magical external defaults that work for
exactly their use case, instead of doing what they should be doing all
along - putting charset right there in the argument.
We need to get people off this mindset fast, since it is not a good one.
Having tons of hidden defaults that modify behavior of functions called
with the same arguments in hundreds of different ways is a coding and
maintenance nightmare. Now if I write `htmlspecialchars()` I can never be
sure if works right and uses UTF-8 - what if somebody messed with the
INI setting because of some other broken library that required that to work?
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #58886 — Rasmus Lerdorf — 2012-03-12T19:44:25+00:00
URL: https://externals.io/message/58853#58886 · In reply to: #58884
> Hi!
>
> > […]
>
> I'm not sure "changing every call" is such a big problem - it's one grep
> and one replace, can be done in one line of sed/awk/perl/php probably.
> But a bigger issue is here that people insist on using wrong charsets
> and expect language to have some magical external defaults that work for
> exactly their use case, instead of doing what they should be doing all
> along - putting charset right there in the argument.
> We need to get people off this mindset fast, since it is not a good one.
> Having tons of hidden defaults that modify behavior of functions called
> with the same arguments in hundreds of different ways is a coding and
> maintenance nightmare. Now if I write `htmlspecialchars()` I can never be
> sure if works right and uses UTF-8 - what if somebody messed with the
> INI setting because of some other broken library that required that to
> work?
But you can't necessarily hardcode the encoding if you are writing
portable code. That's a bit like hardcoding a timezone. In order to
write portable code you need to give people the ability to localize it.
-Rasmus
---
## #58888 — Stas Malyshev — 2012-03-12T19:51:31+00:00
URL: https://externals.io/message/58853#58888 · In reply to: #58886
Hi!
> But you can't necessarily hardcode the encoding if you are writing
> portable code. That's a bit like hardcoding a timezone. In order to
> write portable code you need to give people the ability to localize it.
No, it's not like timezone at all. I have to support all timezones in a
global app, but I don't have to internally support every encoding on
Earth - having everything internally in UTF-8 works quite well, and a
lot of applications do exactly that - they have everything internally in
UTF-8 and only may convert when importing or exporting the data. I don't
see anything in using UTF-8 throughout the app/library that makes it
non-portable. However, if we allow to change defaults in
`htmlspecialchars()` etc. that essentially makes having defaults useless
as I'd have so explicitly specify UTF-8 each time - otherwise it's a
gamble what encoding I'd actually get.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #58890 — Rasmus Lerdorf — 2012-03-12T20:12:03+00:00
URL: https://externals.io/message/58853#58890 · In reply to: #58888
> Hi!
>
> > […]
>
> No, it's not like timezone at all. I have to support all timezones in a
> global app, but I don't have to internally support every encoding on
> Earth - having everything internally in UTF-8 works quite well, and a
> lot of applications do exactly that - they have everything internally in
> UTF-8 and only may convert when importing or exporting the data. I don't
> see anything in using UTF-8 throughout the app/library that makes it
> non-portable. However, if we allow to change defaults in
> `htmlspecialchars()` etc. that essentially makes having defaults useless
> as I'd have so explicitly specify UTF-8 each time - otherwise it's a
> gamble what encoding I'd actually get.
If everything was UTF-8 we wouldn't have any of these issues.
Unfortunately that isn't the case. The question is what to do with apps
that need to deal with non UTF-8 data. Are we going to provide any help
to them beyond just telling them to convert everything to UTF-8?
We took steps in 5.4 to improve htmlspecialchars to understand more
encodings and we have the concept of script_encoding and
internal_encoding that is used both in the engine and in mbstring.
Currently internal_encoding isn't checked by htmlspecialchars. If you
pass it '' it checks script_encoding and default_charset which is a bit
odd since neither directly relate to the encoding of the internal data
you are feeding to it. So maybe a way to tackle this is to use the
mbstring internal encoding when it is set as the htmlspecialchars
default when it is called without an encoding arg.
-Rasmus
---
## #58892 — Pierre Joye — 2012-03-12T20:19:59+00:00
URL: https://externals.io/message/58853#58892 · In reply to: #58890
hi Rasmus,
> If everything was UTF-8 we wouldn't have any of these issues.
> Unfortunately that isn't the case. The question is what to do with apps
> that need to deal with non UTF-8 data. Are we going to provide any help
> to them beyond just telling them to convert everything to UTF-8?
That's not really an acceptable solution, obviously.
> We took steps in 5.4 to improve htmlspecialchars to understand more
> encodings and we have the concept of script_encoding and
> internal_encoding that is used both in the engine and in mbstring.
>
> Currently internal_encoding isn't checked by htmlspecialchars. If you
> pass it '' it checks script_encoding and default_charset which is a bit
> odd since neither directly relate to the encoding of the internal data
> you are feeding to it. So maybe a way to tackle this is to use the
> mbstring internal encoding when it is set as the htmlspecialchars
> default when it is called without an encoding arg.
That's why I would prefer to use an existing setting and clearly
document it instead of creating a new ini settings with a totally
different impact than the existing ones. Not sure which one would fit
best tho'.
Reading these last two paragraphs gave me a headache and I did not
know anymore which encoding we were talking about ;-)
Cheers,
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
---
## #58894 — keisial@gmail.com — 2012-03-12T21:26:36+00:00
URL: https://externals.io/message/58853#58894 · In reply to: #58888
> Hi!
>
> > […]
>
> No, it's not like timezone at all. I have to support all timezones in
> a global app, but I don't have to internally support every encoding on
> Earth - having everything internally in UTF-8 works quite well, and a
> lot of applications do exactly that - they have everything internally
> in UTF-8 and only may convert when importing or exporting the data. I
> don't see anything in using UTF-8 throughout the app/library that
> makes it non-portable. However, if we allow to change defaults in
> `htmlspecialchars()` etc. that essentially makes having defaults useless
> as I'd have so explicitly specify UTF-8 each time - otherwise it's a
> gamble what encoding I'd actually get.
If you are a framework developer, and really want to shield against a
bad php.ini setting, you could `ini_set()` to your prefered charset at the
beginning of the request.
---
## #58895 — Stas Malyshev — 2012-03-12T21:30:28+00:00
URL: https://externals.io/message/58853#58895 · In reply to: #58894
Hi!
> If you are a framework developer, and really want to shield against a
> bad php.ini setting, you could `ini_set()` to your prefered charset at the
> beginning of the request.
That assuming "the request" is completely processed by your framework
and you never call any outside code and any outside code never calls you
- otherwise your messing with INI setting may very well break that code
or that code's messing with INI settings may very well break yours.
--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
---
## #58906 — keisial@gmail.com — 2012-03-12T22:57:42+00:00
URL: https://externals.io/message/58853#58906 · In reply to: #58895
> Hi!
>
> > […]
>
> That assuming "the request" is completely processed by your framework
> and you never call any outside code and any outside code never calls
> you - otherwise your messing with INI setting may very well break that
> code or that code's messing with INI settings may very well break yours.
Sure. That's a setting to be kept the same for the request unless you
like trouble.
If you need to call a library function which uses a different html
charset convention you could do so through a wrapper, which sets and
restores the setting.
Still, that API is likely wrong: a library function written by someone
completely unrelated to the main application shouldn't be echoing
anything through the output. And if it's not generating the html, the
htmlspecialchars is better done from the return at the calling
application (probably after converting the internal charset).
Such interfaces may be well served by switching the setting many times.
I was only advocating the usage of `ini_set()` once in the request,
for the case of a server with two applications having different needs
(equivalent to configuring it on .user.ini or .htaccess).
---
Next page: https://externals.io/message/58853.md?page=2