Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125912
Feedback-ID: i8414410d:Fastmail
Precedence: bulk
MIME-Version: 1.0
Date: Tue, 05 Nov 2024 11:29:02 -0600
To: "php internals" <internals@lists.php.net>
Message-ID: <af567362-efd0-4e4a-9869-2f379f85e0ff@app.fastmail.com>
In-Reply-To: 
 <AM8P250MB0170E88F2B19CA8C91786F66E2562@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
References: <55320aad-758a-4d06-b1bd-3eac2b5a5f71@app.fastmail.com>
 <AM8P250MB0170E88F2B19CA8C91786F66E2562@AM8P250MB0170.EURP250.PROD.OUTLOOK.COM>
Subject: Re: [PHP-DEV] [RFC] PHP.net analytics
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From: larry@garfieldtech.com ("Larry Garfield")

On Fri, Nov 1, 2024, at 6:10 PM, Bob Weinand wrote:
> On 1.11.2024 22:41:29, Larry Garfield wrote:
>> In a similar vein to approving the use of software, Roman Pronskiy as=
ked for my help putting together an RFC on collecting analytics for PHP.=
net.
>>
>> https://wiki.php.net/rfc/phpnet-analytics
>>
>> Of particular note:
>>
>> * This is self-hosted, first-party only.  No third parties get data, =
so no third parties can do evil things with it.
>> * There is no plan to collect any PII.
>> * The goal is to figure how how to most efficiently spend Foundation =
money improving php.net, something that is sorely needed.
>>
>> Ideally we'd have this in place by the 8.4 release or shortly thereaf=
ter, though I realize that's a little tight on the timeline.
>
> Hey Larry,
>
> I have a couple concerns and questions:
>
> Is there a way to track analytics with only transient data? As in, dat=
a=20
> actually stored is always already anonymized enough that it would be=20
> unproblematic to share it with everyone?
> Or possibly, is there a retention period for the raw data after which=20
> only anonymized data remains?

The plan is to configure Matomo to not collect anything non-anonymous to=
 begin with, to the extent possible.  We're absolutely not talking about=
 user-stalking like ad companies do, or anything even remotely close to =
that.

I'm not convinced that publishing raw, even anonymized data, is valuable=
 or responsible.  I don't know of any other sites off hand that publish =
their raw analytics, and I don't know what purpose that would serve othe=
r than just a principled "radical transparency" stance, which I generall=
y don't agree with.

However, having an automated aggregate dashboard similar to https://anal=
ytics.bookstackapp.com/bookstackapp.com (made by a different tool, but s=
ame idea) that we could make public is the goal, but we don't want to do=
 that until it's been running a while and we're sure that nothing person=
ally identifiable could leak through that way.

> Do you actually have a plan what to use that data for? The RFC mostly=20
> talks about "high traffic". But does that mean anything? I do look at =
a=20
> documentation page, because I need to look something specific up (what=20
> was the order of arguments of strpos again?). I may only look shortly =
at=20
> it. Maybe even often. But it has absolutely zero signal on whether the=20
> documentation page is good enough. In that case I don't look at the=20
> comments either. Comments are something you rarely look at, mostly the=20
> first time you want to even use a function.

Right now, the key problem is that there's a lot of "we don't know what =
we don't know."  We want to improve the site and docs, the Foundation wa=
nts to spend money on doing so, but other than "fill in the empty pages"=
 we have no definition of "improve" to work from.  The intent is that be=
tter data will give us a better sense of what "improve" even means. =20

It would also be useful for marketing campaigns, even on-site.  Eg, if w=
e spend the time to write a "How to convince your boss to use PHP" page.=
.. how useful is it?  From logs, all we could get is page count.  That's=
 it.  Or the PHP-new-release landing page that we've put up for the last=
 several releases.  Do people actually get value of that?  Do they bothe=
r to scroll down through each section or do they just look at the first =
one or two and leave, meaning the time we spent on any other items is wa=
sted?  Right now, we have no idea if the time spent on those is even use=
ful. =20

Another example off the top of my head: Right now, the enum documentatio=
n is spread across a dozen sub-pages.  I don't know why I did that exact=
ly in the first place rather than one huge page, other than "huge pages =
bad."  But are they bad?  Would it be better to combine enums back into =
fewer pages, or to split the visibility long-page up into smaller ones? =
 I have no idea.  We need data to answer that.

It's also absolutely true that analytics are not the end of data collect=
ion.  User surveys, usability tests, etc. are also highly valuable, and =
can get you a different kind of data.  We should likely do those at some=
 point, but that doesn't make automated analytics not useful.


Another concern with just using raw logs is that it would be more work t=
o setup, and have more moving parts to break.  Let's be honest, PHP has =
an absolutely terrible track record when it comes to keeping our moving =
parts working, and the Infra Team right now is tiny.  The bus factor the=
re is a concern.  Using a client-side tracker is the more-supported and =
fewer-custom-scripts approach, which makes it easier for someone new to =
pick it up when needed.

Logs also will fold anyone behind a NAT together into a single IP, and t=
hus "user."  IP address is in general a pretty poor way of uniquely iden=
tifying people with the number of translation layers on the Internet the=
se days.

> Overall I feel like the signal we can get from using a JS tracker=20
> specifically is comparatively low to the point it's not actually worth=
 it.

Some more things a client-side tracker could do that logs cannot:

* How many people are accessing the site from a desktop vs mobile?
* What speed connection do people have?
* How many people are using the in-browser Wasm code runner that is curr=
ently being worked on?  cf: https://github.com/php/web-php/pull/1097

Also, for reference, most language sites do have some kind of analytics,=
 usually Google:

https://www.python.org =E2=80=93Plausible.io, Google analytics
https://go.dev/ =E2=80=94 Google Analytics
https://www.rust-lang.org/ =E2=80=93N/A
https://nodejs.org/ =E2=80=93 Google Analytics
https://www.typescriptlang.org/ =E2=80=93 N/A
https://kotlinlang.org/  =E2=80=93 Google Analytics
https://www.swift.org/  =E2=80=93 Adobe Analytics
https://www.ruby-lang.org/  =E2=80=93 Google Analytics

We'd be the only one with a self-hosted option, making it the most priva=
cy-conscious of the bunch.

As far as blocking the analytics goes, Matomo uses a cookieless approach=
, so it's rarely blocked (and would not need a GDPR-compliance banner). =
 Even if someone wanted to block it, meh.  We'd still be getting enough =
signal to make informed decisions.

--Larry Garfield