Newsgroups: php.internals Path: news.php.net Xref: news.php.net php.internals:55341 Return-Path: Mailing-List: contact internals-help@lists.php.net; run by ezmlm Delivered-To: mailing list internals@lists.php.net Received: (qmail 70972 invoked from network); 9 Sep 2011 17:39:08 -0000 Received: from unknown (HELO lists.php.net) (127.0.0.1) by localhost with SMTP; 9 Sep 2011 17:39:08 -0000 Authentication-Results: pb1.pair.com smtp.mail=nikita.ppv@googlemail.com; spf=pass; sender-id=pass Authentication-Results: pb1.pair.com header.from=nikita.ppv@googlemail.com; sender-id=pass Received-SPF: pass (pb1.pair.com: domain googlemail.com designates 209.85.215.176 as permitted sender) X-PHP-List-Original-Sender: nikita.ppv@googlemail.com X-Host-Fingerprint: 209.85.215.176 mail-ey0-f176.google.com Received: from [209.85.215.176] ([209.85.215.176:58167] helo=mail-ey0-f176.google.com) by pb1.pair.com (ecelerity 2.1.1.9-wez r(12769M)) with ESMTP id 69/77-33036-B3F4A6E4 for ; Fri, 09 Sep 2011 13:39:08 -0400 Received: by eyz10 with SMTP id 10so1662071eyz.35 for ; Fri, 09 Sep 2011 10:39:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PhYNGfO3KBPN9hWi5js0vsHIz79RS+kEwyGMNCbq0uI=; b=mfpsIjun8U9IroxBoPsMer9/p1qs6alA3dEPgQSLUJ6+xt3S7axVCPOYdvX9RtfMEo nShYGZ7M4nzISmLQEmPuuEkpIlTiz3A/VQpq1wrVrI0eL7y7fH7Hs80lB7de+Xsdp7ES v1rHckRJ+wqf+BExTMtK4X7ucijGiqzgnEHok= MIME-Version: 1.0 Received: by 10.14.34.9 with SMTP id r9mr738773eea.142.1315589943635; Fri, 09 Sep 2011 10:39:03 -0700 (PDT) Received: by 10.14.96.136 with HTTP; Fri, 9 Sep 2011 10:39:03 -0700 (PDT) In-Reply-To: References: Date: Fri, 9 Sep 2011 19:39:03 +0200 Message-ID: To: Nicolas Grekas Cc: Ferenc Kovacs , PHP internals Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: [PHP-DEV] Revert Tokenizer behavior for 5.4 From: nikita.ppv@googlemail.com (Nikita Popov) On Fri, Sep 9, 2011 at 10:46 AM, Nicolas Grekas wrote: > You can also just count the number of semantic token after T_HALT_COMPILER > (ie excluding whitespace and comments) and once you hit 3, halt. > [...] > Maybe the binary data itself, as a big T_INLINE_HTML for example ? In favor of both proposals! Returning the next 3 tokens should be quite easy [1]. Returning the rest as an T_INLINE_HTML makes sense too, as extracting the data is probably what you want. Though I have no idea how to implement that ^^ [1]: https://github.com/nikic/php-src/commit/2d4cfa05947f04de447635ca5748b3b58defbfaf (Not tested, only guessing)