Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:117061
Message-ID: <26a8c3ee-9f0a-793c-10c0-7e642eedf1d0@bastelstu.be>
Date: Fri, 18 Feb 2022 11:46:13 +0100
MIME-Version: 1.0
Content-Language: en-US
To: Go Kudo <g-kudo@colopl.co.jp>
Cc: internals@lists.php.net
References: <CAMs36HcsGqNanQaro_VTkK_VKHNg-jqtshXpW-aCxdqpxHjMdw@mail.gmail.com>
 <41a1b458-4941-f34e-f1b4-e25b3298b80a@bastelstu.be>
 <CAMs36Hc4GCPUp2m+6N1LTP+hPrUv15iZF2QTCHBuAm+KDyXgfw@mail.gmail.com>
 <553ba7ca-3821-c2d9-f88f-b216013a887b@bastelstu.be>
 <a2683304-542f-e63a-e35f-04e5bb75f5d1@bastelstu.be>
 <CAMs36Hc+W_GkO4W+FB=ySFvc79DYy+M3TaT5qL0Ap9NwmY=6dQ@mail.gmail.com>
 <2c667812-88c8-0b7b-3558-561a1348d0b2@bastelstu.be>
 <CAMs36HctoKykr=aSVCr_DtFEpeF1fZke5LGCOFu_-oQEzL06Ug@mail.gmail.com>
 <a94b1b01-dd9c-bb62-01ab-34c16c3d46cb@bastelstu.be>
 <CAMs36HcVYNVB+mukx27Z90+5DtiHz1JO0i7qF+kP6UaML-y3ZQ@mail.gmail.com>
 <5f496cf9-8754-b009-9cb5-b978222b2249@bastelstu.be>
 <CAMs36HdSf8Mh9Mugwvb643xNfLnpZQ0RAvV_ZuS11vQzQRTmZA@mail.gmail.com>
 <cc43d1ce-8807-c0fd-94a4-514ebedea661@bastelstu.be>
 <CAMs36HcC7O7YR6VVybw=ChPrWWcWjnCTxR6_=5scFy0+ksZEaA@mail.gmail.com>
 <a1369ca5-9d27-8151-300d-73b90cd584d3@bastelstu.be>
 <CAMs36HcLgJszZ2LuUKjuykr4uLNP0+LdJd6disx_Xnbs+K63Xw@mail.gmail.com>
In-Reply-To: <CAMs36HcLgJszZ2LuUKjuykr4uLNP0+LdJd6disx_Xnbs+K63Xw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] [RFC] [Under Discussion] Random Extension 4.0
From: tim@bastelstu.be (=?UTF-8?Q?Tim_D=c3=bcsterhus?=)

Hi

On 2/18/22 07:31, Go Kudo wrote:
> I have been looking into output buffering, but don't know the right way to
> do it. The buffering works fine if all RNG generation widths are static,
> but if they are dynamic so complicated.

I believe the primary issue here is that the engines are expected to 
return an uint64_t, instead of a buffer with raw bytes. This requires 
you to perform many conversions between the uint64 and the raw buffer:

When calling Randomizer::getBytes() for a custom engine the following 
needs to happen:

- The Engine returns a byte string.
- This bytestring is then internally converted into an uint64_t.
- Then calling Randomizer::getBytes() this uint64_t needs to be 
converted back to a bytestring.

To avoid those conversations without sacrificing too much performance it 
might be possible to return a struct that contains a single 4 or 8-byte 
array:

     struct four_bytes {
         unsigned char val[4];
     };

     struct four_bytes r;
     r.val[0] = (result >> 0) & 0xff;
     r.val[1] = (result >> 8) & 0xff;
     r.val[2] = (result >> 16) & 0xff;
     r.val[3] = (result >> 24) & 0xff;

     return r;

.val can be treated as a bytestring, but it does not require dynamic 
allocation. By doing that the internal engines (e.g. Xoshiro) would be 
consistent with the userland engines.

> It is possible to solve this problem by allowing generate() itself to
> specify the size it wants, but this would significantly slow down
> performance.

I don't think it's a good idea to add a size parameter to generate().

> I've looked at the sample code, but do you really need support for
> Randomizer? Engine::generate() can output dynamic binaries up to 64 bits.
> You can use Engine directly, instead of Randomizer::getBytes().
> 
> What exactly is the situation where buffering by Randomizer is needed?

*I* don't need anything. I'm just trying to think of use-cases and 
edge-cases. Basically: What would a user attempt to do and what would 
their expectations be?

I'm not saying that this buffering *must* be implemented, but this is 
something we need to think about. Because changing the behavior later is 
pretty much impossible, as users might rely on a specific behavior for 
their seeded sequences. The behavior might also need to be part of the 
documentation.

Basically what we need to think about is what guarantees we give. As an 
example:

1. Calling Engine::generate() with the same seed results in the same 
sequence (This guarantee we give, and it is useful).
2. Calling Randomizer::getInt() with the same seeded engine results in 
the same numbers for the same parameters (I think this also is useful).
3. Calling Randomizer::getBytes() with the same seeded engine results in 
the same byte sequence (This is something we are currently discussing).
4. Calling Randomizer::getBytes() simply concatenates the raw bytes 
retrieved by the Engine (This ties into (3)).
5. Calling Randomizer::shuffleArray() with the same seeded engine 
results in the same result for the same string (This one is more 
debatable, because then we must maintain the exact same shuffleArray() 
implementation forever).

All these guarantees should be properly documented within the RFC. The 
RFC template (https://wiki.php.net/rfc/template) says:

 >  Remember that the RFC contents should be easily reusable in the PHP 
Documentation.

So by thinking about this now and putting it in the RFC, the 
explanations can easily be copied into the documentation if the RFC 
passes the vote.

One should not need to look into the implementation to understand how 
the Engines and the Randomizer is supposed to work.

> Also worried that buffering will cut off random numbers at arbitrary sizes.
> It may cause bias in the generated results.
> 

If there's bias in specific bits or bytes of the generated number then 
getBytes(32) will already be biased even without buffering, as the raw 
bytes are what's of interest here. It does not matter if they are at the 
1st or 4th position (for a 32-bit engine).

Best regards
Tim Düsterhus