Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:43641
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: unknown (pb1.pair.com: domain php.net does not designate 208.43.138.18 as permitted sender)
Message-ID: <49DA823A.509@php.net>
Date: Mon, 06 Apr 2009 15:29:14 -0700
Reply-To: shire@php.net
User-Agent: Postbox 1.0b10 (Macintosh/2009032714)
MIME-Version: 1.0
To: Matt Wilmas <php_lists@realplain.com>
CC: php-dev List <internals@lists.php.net>, 
 Lukas Kahwe Smith <mls@pooteeweet.org>,
 Pierre Joye <pierre.php@gmail.com>, Dmitry Stogov <dmitry@php.net>, 
 Christian Seiler <chris_se@gmx.net>
References: <B3E41C55-0E3A-4EF6-B11A-E14AFE438204@pooteeweet.org> <fe05d1540903301527n312e8e5dy4f99f490283662e@mail.gmail.com> <90FF707C-391F-4FF4-8C5E-2AECEB48EB78@pooteeweet.org> <7C741E86-7720-4C87-8A43-BCC34520F0B5@pooteeweet.org> <0729A9B70D864C6E929169523CEBCB11@pc1> <49DA38C7.10006@php.net> <08D45CD5FF6A4A30BA67DC2D50CB5F52@pc1>
In-Reply-To: <08D45CD5FF6A4A30BA67DC2D50CB5F52@pc1>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] RC2 and integer/float handling in 5.3
From: shire@php.net (shire)


Hey Matt,

Matt Wilmas wrote:
> Yep, 5.3's snapshot self-compiled from a couple days ago on Windows (not
> that that should matter). (I'm not regenerating it with re2c, which also
> shouldn't matter; using the existing .c file. I haven't touched the
> scanner stuff in a long time (yet) to regen.) Scanner of course hasn't
> changed since then.

Here's what I'm currently doing (more or less with some changed paths):

$ echo $PHPCVS
:pserver:shire@cvs.php.net:/repository

$ cvs -d $PHPCVS co -r PHP_5_3 php5
...

$ cd php5

$ ./buildconf && ../config.nice && make
...

$ cat ../bug46817-1.php
<?php

var_dump(token_get_all(file_get_contents('../bug46817-2.php')));

$ cat ../bug46817-2.php
<?php
// this comment and trailing blank contain windows CR+LF


$ ./sapi/cli/php ../bug46817-1.php
array(3) {
   [0]=>
   array(3) {
     [0]=>
     int(368)
     [1]=>
     string(6) "<?php
"
     [2]=>
     int(1)
   }
   [1]=>
   array(3) {
     [0]=>
     int(366)
     [1]=>
"   string(57) "// this comment and trailing blank contain windows CR+LF
     [2]=>
     int(2)
   }
   [2]=>
   array(3) {
     [0]=>
     int(371)
     [1]=>
     string(3) "

"
     [2]=>
     int(2)
   }
}


The newlines look like this in the second file:

<?php$
// this comment and trailing blank contain windows CR+LF^M$
^M$

Unfortunately I can't test on a windows build, perhaps you could re-test or share your reproduction that fails as this seems to work for me unless I'm of course missing some difference.


> Test case is the one in the bug report. :-) Last token is not the
> comment, but whitespace.

There are two reproductions in the bug report ;-)



> Also, the unterminated comment Warning is still missing with "<?php /*
> blah " like it's been since the re2c change (except maybe for the time
> your fix was applied). My changes would clean this up of course, unless
> you do something first.

I think fixing this would be great as well as the other highlighter test that was changed.  I would just prefer that the scanner handle these rather than us implementing what is essentially a hand-written scanner within the lexer file.


-shire