# Shifting bits of a binary string

- Thread: https://externals.io/message/73681
- Original archive: https://news-web.php.net/php.internals/73681
- Started by Marc Bennewitz on 2014-04-13T12:20:49+00:00
- Messages: 7
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #73681 — Marc Bennewitz — 2014-04-13T12:20:49+00:00

URL: https://externals.io/message/73681#73681

Hi List,

I hope I'm on the right list but I can't find any other helpful.

I have a binary string and I would like to work with bitwise operators.
The only help I found was to convert it to an integer. That's ok but it 
results in some questions:

  - What if the binary data is more than 32/64 bits long?
  - Why converting binary data of form one into binary data of another 
form only to manipulate bits?

So I simply tested what's going on if I operate on a string directly but 
on shifting I get the same wrong result every time.
(Testscript below)

On reading the manual the only note for strings are the following:
(http://www.php.net/manual/en/language.operators.bitwise.php)
 > Be aware of data type conversions. If both the left-hand and 
right-hand parameters are strings, the bitwise operator will operate on 
the characters' ASCII values.

Why such bit operators doesn't work with strings?
Why there is not helpful information about in the manual.
Why on operation something not working doesn't result in an error/notice 
but in a completely unexpected value?

Greetings
Marc


Shift to the left:
var_dump(decbin(ord(chr(1))));
for ($i=0; $i<10; $i++) {
     var_dump(decbin(ord(chr(1) << $i)));
}

Output:
string(1) "1"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"

Shift to the right:
var_dump(decbin(ord(chr(32))));
for ($i=0; $i<10; $i++) {
     var_dump(decbin(ord(chr(32) >> $i)));
}

Output:
string(1) "100000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"
string(6) "110000"

---

## #73685 — Chris Wright — 2014-04-14T11:12:16+00:00

URL: https://externals.io/message/73681#73685 · In reply to: #73681

Hi Marc


> Hi List,
>
> I hope I'm on the right list but I can't find any other helpful.
>
> I have a binary string and I would like to work with bitwise operators.
> The only help I found was to convert it to an integer. That's ok but it
> results in some questions:
>
>  - What if the binary data is more than 32/64 bits long?
>  - Why converting binary data of form one into binary data of another form
> only to manipulate bits?
>
> So I simply tested what's going on if I operate on a string directly but on
> shifting I get the same wrong result every time.
> (Testscript below)
>
> On reading the manual the only note for strings are the following:
> (http://www.php.net/manual/en/language.operators.bitwise.php)
> > […]
>
> Why such bit operators doesn't work with strings?
> Why there is not helpful information about in the manual.
> Why on operation something not working doesn't result in an error/notice but
> in a completely unexpected value?
>
> Greetings
> Marc
>
>
> Shift to the left:
> var_dump(decbin(ord(chr(1))));
> for ($i=0; $i<10; $i++) {
>     var_dump(decbin(ord(chr(1) << $i)));
> }
>
> Output:
> string(1) "1"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
>
> Shift to the right:
> var_dump(decbin(ord(chr(32))));
> for ($i=0; $i<10; $i++) {
>     var_dump(decbin(ord(chr(32) >> $i)));
> }
>
> Output:
> string(1) "100000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"
> string(6) "110000"

First, an explanation of why you see the results you show here:

At present, the shift operations act on long integers, when either
operand is not an integer, they converted to integers. In the case of
strings, this means they are passed though strtol() with an explicit
base of 10, meaning that in your example above (and for any string
that is not a decimal integer) the result of the conversion will be
zero. The operation will then be performed with a left operand of
zero, so the result will also be zero.

When your code dumps this you pass it through `ord()`, which (via a zpp
call) converts the integer to a string, and then converts the first
character of this string back to an integer, resulting in 48, the
ordinal value of ASCII "0". You then pass it to `decbin()`, which
returns the binary representation of 48.

The following gives a result that may be more like what you would expect:

$base = '1';
var_dump(decbin($base));
for ($i=0; $i<10; $i++) {
    var_dump(decbin($base << $i));
}

With regards to the actual issue, this is something I would also like
to see "fixed".

There is an issue with bitwise operations on strings, and that is that
they are not as cheap as they are with integers - people may expect
bitwise operations to be lightweight wrappers around very basic
processor instructions, and in that case of strings this is not true
because the operation must be performed byte-by-byte in a O(n) manner.
With shifts the functional complexity further increases, as there is
additional branching required as often bits must be carried between
bytes, in which case each byte must be visited twice.

None of these issues actually prevent this from being possible though,
and while the use cases case for this are few and far between I think
the current behaviour unexpected and not the sensible option.

*However* there is a very real BC issue here. Consider some code that
relies on the result of $_GET['mask'] << 2 or something similar -
something that I can imagine someone somewhere as done, and will break
if the behaviour is "fixed". Anywhere that input is collected is it
usually present as a string, and the current behaviour allows you to
treat it as an integer and get the result you expect.

 I would argue that this person did it wrong in the first place and
that they should be paying attention to types if they are performing
bitwise operations. I would also be happy to break BC on this part of
the language that I doubt is used very often. But at the end of the
day what really matters is what everyone thinks, not just what I
think. I would be surprised if this hasn't been previously discussed
on the list - I know I've had a few discussions on the subject with
various people off-list over the last year or two.

I will try and throw a patch together at lunch to give the behaviour I
would expect by means of special-case handling for a left operand of
type IS_STRING, but it is a BC break that would probably only be
accepted into 5++, if at all.

Thanks, Chris

---

## #73689 — Marc Bennewitz — 2014-04-14T18:52:27+00:00

URL: https://externals.io/message/73681#73689 · In reply to: #73685

Hi Chris,

Thanks a lot for your detailed explanation!

For a way forward to get it "fixed" the PHP Community needs to find out 
what of the current behaviors (not only this one) are bugs and what are 
features I think and how to handle changes in each case. But that's 
another discussion. (I like the "use strict" idea)

In the case of bit shifting I would expect the following but this is 
only my point of view:

- Bit shifting on integers:
   - Same as current behavior but don't preserve the first bit
     or PHP needs an unsigned integer type - I would love it :)

- Bit shift left on strings:
   - act byte-by-byte in a O(n) manner
   - shifting out left bit(s)
   - append `NULL` bits to the right
   - preserve string length

- Bit shift right on strings:
   - act byte-by-byte in a O(n) manner
   - shifting out right bit(s)
   - prepend `NULL` bits to the left (don't preserve the first bit)
   - preserve string length

- Bit shifting on other types:
   - error/warning

- If the number of bits to shift isn't an integer, than:
   - error


Only my 2 cents
Marc


> Hi Marc
>
> 
> > […]
>
> First, an explanation of why you see the results you show here:
>
> At present, the shift operations act on long integers, when either
> operand is not an integer, they converted to integers. In the case of
> strings, this means they are passed though strtol() with an explicit
> base of 10, meaning that in your example above (and for any string
> that is not a decimal integer) the result of the conversion will be
> zero. The operation will then be performed with a left operand of
> zero, so the result will also be zero.
>
> When your code dumps this you pass it through `ord()`, which (via a zpp
> call) converts the integer to a string, and then converts the first
> character of this string back to an integer, resulting in 48, the
> ordinal value of ASCII "0". You then pass it to `decbin()`, which
> returns the binary representation of 48.
>
> The following gives a result that may be more like what you would expect:
>
> $base = '1';
> var_dump(decbin($base));
> for ($i=0; $i<10; $i++) {
>      var_dump(decbin($base << $i));
> }
>
> With regards to the actual issue, this is something I would also like
> to see "fixed".
>
> There is an issue with bitwise operations on strings, and that is that
> they are not as cheap as they are with integers - people may expect
> bitwise operations to be lightweight wrappers around very basic
> processor instructions, and in that case of strings this is not true
> because the operation must be performed byte-by-byte in a O(n) manner.
> With shifts the functional complexity further increases, as there is
> additional branching required as often bits must be carried between
> bytes, in which case each byte must be visited twice.
>
> None of these issues actually prevent this from being possible though,
> and while the use cases case for this are few and far between I think
> the current behaviour unexpected and not the sensible option.
>
> *However* there is a very real BC issue here. Consider some code that
> relies on the result of $_GET['mask'] << 2 or something similar -
> something that I can imagine someone somewhere as done, and will break
> if the behaviour is "fixed". Anywhere that input is collected is it
> usually present as a string, and the current behaviour allows you to
> treat it as an integer and get the result you expect.
>
>   I would argue that this person did it wrong in the first place and
> that they should be paying attention to types if they are performing
> bitwise operations. I would also be happy to break BC on this part of
> the language that I doubt is used very often. But at the end of the
> day what really matters is what everyone thinks, not just what I
> think. I would be surprised if this hasn't been previously discussed
> on the list - I know I've had a few discussions on the subject with
> various people off-list over the last year or two.
>
> I will try and throw a patch together at lunch to give the behaviour I
> would expect by means of special-case handling for a left operand of
> type IS_STRING, but it is a BC break that would probably only be
> accepted into 5++, if at all.
>
> Thanks, Chris

---

## #73692 — Chris Wright — 2014-04-15T00:04:04+00:00

URL: https://externals.io/message/73681#73692 · In reply to: #73689

> - Bit shift left on strings:
>   - act byte-by-byte in a O(n) manner
>   - shifting out left bit(s)
>   - append `NULL` bits to the right
>   - preserve string length
>
> - Bit shift right on strings:
>   - act byte-by-byte in a O(n) manner
>   - shifting out right bit(s)
>   - prepend `NULL` bits to the left (don't preserve the first bit)
>   - preserve string length

Agreed.

> - Bit shifting on other types:
>   - error/warning

Probably agreed.

> - If the number of bits to shift isn't an integer, than:
>   - error

Floats would also need to be permitted here, albeit converted to
integers for the purposes of performing the operation.

>
> Only my 2 cents
> Marc
>
>
> 
> > […]

---

## #73712 — Marc Bennewitz — 2014-04-17T09:06:41+00:00

URL: https://externals.io/message/73681#73712 · In reply to: #73692

> 
> > […]
>
> Agreed.
>
> > […]
>
> Probably agreed.
>
> > […]
>
> Floats would also need to be permitted here, albeit converted to
> integers for the purposes of performing the operation.
Ho do you handle values like 3.5? Is it an error if fractions exists?

>
> > […]

---

## #73713 — Chris Wright — 2014-04-17T09:22:14+00:00

URL: https://externals.io/message/73681#73713 · In reply to: #73712

>
>
> 
> > […]
>
> Ho do you handle values like 3.5? Is it an error if fractions exists?

This would be handled as it is currently, with a standard integer
conversion, i.e. rounded towards zero. It doesn't make sense to check
for fractions and error out IMO, as floating point precision errors
would likely cause someone somewhere some unexpected behaviour. In the
case of float(3.5) it's obvious this isn't "correct", but in the case
of float(3.005) the user probably did some calculation that they
expected to result in int(3), so emitting an error would be
unexpected. While a margin of error threshold could be defined, it's
an extra branch and a somewhat arbitrary limitation.

>
>
> > […]

---

## #74235 — Marc Bennewitz — 2014-05-15T19:10:01+00:00

URL: https://externals.io/message/73681#74235 · In reply to: #73713

sorry for the late response.


> 
> > […]
>
> This would be handled as it is currently, with a standard integer
> conversion, i.e. rounded towards zero. It doesn't make sense to check
> for fractions and error out IMO, as floating point precision errors
> would likely cause someone somewhere some unexpected behaviour. In the
> case of float(3.5) it's obvious this isn't "correct", but in the case
> of float(3.005) the user probably did some calculation that they
> expected to result in int(3), so emitting an error would be
> unexpected. While a margin of error threshold could be defined, it's
> an extra branch and a somewhat arbitrary limitation.

After thinking more about on operating bit operations a developer have 
to know about how data are represented in bits. That means the datatype 
to run bit operations on should never be converted automatically only an 
error/warning should be generated if it's not possible to run bit 
operations on it (like objects).

That also means first bit should not be preserved as it was implemented 
to preserve signs of numbers.

And more ... bit shifting of floating point numbers should simply be 
done. Because it makes little sense to do that an error could be 
discussable but in some special cases it could make good sense. (This 
would require to define data structure of floating point numbers 
"double" to be the same 100% on all systems)

Greetings
Marc