Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:102939
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain lerdorf.com designates 209.85.218.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CACXBjujgbzX5AHJzO+6HrMTt0=yqXz0XOxZuVCThjrKaj2ntew@mail.gmail.com>
References: <CACXBjuh01Optb_LBXw899-+i+fM9iEwfOh4zjSHkOjhPJkoD_g@mail.gmail.com>
 <CAPv8svWyfqXB_yS8T24nL=ibxt3n7fBg0nC4GWN0h9agwDAxmw@mail.gmail.com>
 <CANUQDCic9DdUL-a7JjjaZq+QL3TH7=WOn9GmYyJTuYbWQdONCQ@mail.gmail.com>
 <3ce44a21a935f3d458bd4fea99db89a4fd2c9603.camel@ku.edu> <CACXBjujgbzX5AHJzO+6HrMTt0=yqXz0XOxZuVCThjrKaj2ntew@mail.gmail.com>
Date: Sat, 21 Jul 2018 08:30:15 -0400
Message-ID: <CACXBjujUzEEnVg_YJvfOjsJ2sruGqNL+dGojg+gDvMvdPvRh-w@mail.gmail.com>
To: "Hoffman, Zachary Robert" <zrhoffman@ku.edu>
Cc: "mapopa@gmail.com" <mapopa@gmail.com>, "me@kelunik.com" <me@kelunik.com>, 
	"internals@lists.php.net" <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="00000000000071693a057181952c"
Subject: Re: [PHP-DEV] bugs.php.net downtime
From: rasmus@lerdorf.com (Rasmus Lerdorf)

--00000000000071693a057181952c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

For future reference, here is what I did to fix the encoding problem:

MariaDB [phpbugsdb]> select sdesc from bugdb where id=3D76553;
+--------------------------------------------------------------------------=
---------------------------------------------------------------------------=
---------------+
| sdesc

             |
+--------------------------------------------------------------------------=
---------------------------------------------------------------------------=
---------------+
| =C3=90=CB=9C=C3=90=C2=BC=C3=91 =C3=90=C2=BF=C3=90=C2=B5=C3=91=E2=82=AC=C3=
=90=C2=B5=C3=90=C2=BC=C3=90=C2=B5=C3=90=C2=BD=C3=90=C2=BD=C3=90=C2=BE=C3=90=
=C2=B9 =C3=90=C2=BC=C3=90=C2=BE=C3=90=C2=B6=C3=90=C2=B5=C3=91=E2=80=9A =C3=
=91 =C3=90=C2=BE=C3=90=C2=B4=C3=90=C2=B5=C3=91=E2=82=AC=C3=90=C2=B6=C3=90=
=C2=B0=C3=91=E2=80=9A=C3=91=C5=92 =C3=91=C6=92=C3=90=C2=BF=C3=91=E2=82=AC=
=C3=90=C2=B0=C3=90=C2=B2=C3=90=C2=BB=C3=91
=C3=91=C5=BD=C3=91=E2=80=B0=C3=90=C2=B8=C3=90=C2=B5
                |
+--------------------------------------------------------------------------=
---------------------------------------------------------------------------=
---------------+
1 row in set (0.00 sec)

MariaDB [phpbugsdb]> alter table bugdb drop index email;
Query OK, 76298 rows affected (0.85 sec)
Records: 76298  Duplicates: 0  Warnings: 0

MariaDB [phpbugsdb]> alter table bugdb modify sdesc varbinary(80) NOT NULL
DEFAULT '', modify ldesc binary NOT NULL, modify email varbinary(40) NOT
NULL DEFAULT '';
Query OK, 76298 rows affected, 65535 warnings (0.65 sec)
Records: 76298  Duplicates: 0  Warnings: 76091

MariaDB [phpbugsdb]> alter table bugdb modify sdesc varchar(80) CHARACTER
SET utf8mb4 NOT NULL DEFAULT '', modify ldesc text CHARACTER SET utf8mb4
NOT NULL, modify email varchar(40) CHARACTER SET utf8mb4 NOT NULL DEFAULT
'';
Query OK, 76298 rows affected, 127 warnings (0.57 sec)
Records: 76298  Duplicates: 0  Warnings: 127

MariaDB [phpbugsdb]> alter table bugdb add FULLTEXT INDEX `email`
(`email`,`sdesc`,`ldesc`);
Query OK, 76298 rows affected (1.56 sec)
Records: 76298  Duplicates: 0  Warnings: 0

MariaDB [phpbugsdb]> select sdesc from bugdb where id=3D76553;
+--------------------------------------------------------------------------=
--------+
| sdesc
       |
+--------------------------------------------------------------------------=
--------+
| =D0=98=D0=BC=D1=8F =D0=BF=D0=B5=D1=80=D0=B5=D0=BC=D0=B5=D0=BD=D0=BD=D0=BE=
=D0=B9 =D0=BC=D0=BE=D0=B6=D0=B5=D1=82 =D1=81=D0=BE=D0=B4=D0=B5=D1=80=D0=B6=
=D0=B0=D1=82=D1=8C =D1=83=D0=BF=D1=80=D0=B0=D0=B2=D0=BB=D1=8F=D1=8E=D1=89=
=D0=B8=D0=B5
        |
+--------------------------------------------------------------------------=
--------+
1 row in set (0.00 sec)

The trick was to convert the columns to binary first. When I went straight
from latin1 to utf8 I got the utf8 equivalent of the latin1 characters. By
telling it that the data was actually binary first, it converted from
binary to utf8 which appears to have worked. There were some warnings,
which I assume are invalid utf8 byte sequences somewhere.

-Rasmus

--00000000000071693a057181952c--