Collecting metrics

Meta Marketing
1

I’d like to share some thoughts in regards to start collecting metrics. Especially I would like to write about the Why? we need to collect metrics and finish with a more concrete plans we have in Marketing team regarding this.

Also sorry for the long post, but I think it was time to write all of this down somewhere.

Why?

For every work we do we want to know if we are making any impact.

In marketing team we are asking ourselves the same question. “Are we working on the right things?”, “Is what we did, made the expected impact?”. And we would like to answer this questions.

For that reason we plan to start collecting metrics from different parts of the community and bringing them to one place where we could discuss them.

The big plan

We could go wild and start collecting all sorts of metrics, but - as anything in life - there are limits to what we can do in the time we have available. For this reason it is important to know what we want to measure and focus on collecting metrics that would prove/disprove/monitor our plans.

What we are looking for how is the adoption of Nix going.

So what metrics would we need to measure the adoption of Nix? How would we know where in the adoption process are we?

1. The discovery phase

If you write the smartest piece of code and not talk about it, the adoption is less likely to happen. Any effort, that marketing team will do to share the word about Nix, is important to measure the impact it had.

Metrics that we need to collect and that will measure how good we are at the discovery phase are:

  • Google search trends
  • Social media impressions
  • Visits to the landing page

What we are going to look is a corelation between growing (I hope) number of mentions of Nix/NixOS and see if that translates to increase in visits to the nixos.org landing page.

To improve our current situation in this phase the plan is to:

Improve the website and messaging

Some time ago we started working on a redesign of the website and slightly changing the messaging around on the website. That was only a start of the journey.

We are hoping to improve even further on messaging and give landing page a special care. While I’d like to elaborate more on this topic I promise to come back to it in my next post.

Increase presence on social platforms

When good things happen we should have a way to tell people about it. We need to create a space on social media platforms where we could share our story. But that doesn’t happen over night, it requires care and diligence.

While we would like to be on as many platforms as we can, we are aware of the massive amount of work this will take. We would not like to rely on one man heoism (that leads to burnouts), but rather create a process where managing social platforms is manageble.

For now plan is to gradually add Nix to different platforms, starting with Twitter and adding LinkedIn along the way. Once we ensure that the process is manageble we will also look into other platforms.

2. The user phase

The next phase we should monitor is the user phase. This is where some developer goes from “I heard about Nix” to “I’m using Nix”.

To know if the user base of Nix/NixOS is growing we plan to collect the following metrics:

  • Matrix (number of members, messages)
  • Discord (number of members, messages, voice chat hours)
  • Discourse (number of members, members with > 0 hours read time, posts&topics, total days visited)
  • cache.nixos.org usage
  • search.nixos.org (number of queries from elasticsearch)
  • Website statictics on learning materials

The plan to improve this phase is to:

  • create short summaries about things that are going in Nix community (video release announcements, revive weekly newsletter, …)
  • work on better learning materials. Which should also be a topic to talk about on its own. Once we have some more concrete plans I promise to write about this as well.

3. The contibution phase

The contribution phase is when a user of Nix becomes a contributor. This phase is important to ensue the future of Nix/NixOS.

The metrics we plan to gather are mostly to fetch from GitHub. We plan to collect:

  • Contributors (anyone who has authored a commit that’s been merged to master)
  • Recent contributors (anyone who has authored a commit in the last 30 days)
  • Merged PRs
  • Comments
  • Stars
  • Forks

Conclusion

I think it goes without saying that we will NOT be collecting any personal data in and will be open about the data we collected. Anybody should be able to make a story from the collected data.

The above plan is just the initial draft of the work that we plan to work on at Marketing team. If you think we missed some important metrics or something is not covered by our metrics but it should be we would love to hear you opinion in the comments bellow.

There is a lot of work ahead of us and if you feel like you could I’d welcome you to join the Marketing team matrix channel where we will organize who works on what.

The more technical part of the plan is to use the existing Prometheus and Grafana instance to import and graph all the collected metrics. But this is something to write about next time.

Thank you for you patience and I hope I didn’t do to many typos this time :slight_smile:

11 Likes
2

Will these be available publicly?

I’d also like to point out:

While that seems reasonable enough to me personally, this does mean you will almost certainly be collecting PII data. While I understand that those are also collected by anyone cloning the repository, and literally publicly accessible to anyone on the internet, it’s probably worth figuring out what the EU says about creating a derivative database from that. At the very least, I think the “legitimate interest” claim becomes weaker.

If you do end up collecting such data, it would make sense to collect it for the current stable branch too, perhaps including committers so that backports are captured correctly. It would be interesting to see how the community clusters between unstable/stable use.

3 Likes
3

I think I need to clarify this. We would collect the aggregate of contributors, eg. on day X there were 10 commit by 3 contributors and 1 new contributor. Something along those lines, without any personal data, even if that is already public in git repositories.

But good point in regards to stable branch and backporting commits. Thank you.

4

Love how the phases are divided. This makes sense and I’m very happy to see this work go forward. Go Marketing Team!

1 Like
5

Maybe this can be an inspiration: https://chaoss.community/

2 Likes
6

That’s a bit vague, what kind of metrics are we expecting to collect here?

I think that while collecting aggregated traffic metrics (ie. xGB/h) is fine, anything more refined can quickly become pretty dangerous in terms of user privacy.

I’d be reassured if the marketing team fully qualify what they intend to do in terms of cache metrics.

(just to be explicit: I’m personally fine with the other metrics listed here)

1 Like
7

I don’t the exact details about every metrics just yet, but you can be assured that we will collect some sort of aggregated metrics that already existing services that we are using provide. I assume in the case of cache.nixos.org this would be something from Fastly.

I hope to soon have an update with more details. This mostly is when there will be enough time to work further on this. But this is high on my TODO list.

2 Likes
8

I would love to know which of my packages see the most cache hits, or if any don’t see any use and can be dropped/moved to NUR

For the getting more users angle, some of these stats would be very interesting.

Although I’d add we’ve already got some really good stats from repology we should be amplifying around non-unique package coverage and how up to date our packages are etc.

For socials are there any good tools for aggregating social platforms? Just to make posting easier for the marketing team.

I know back in the day hootsuite was popular. Socioboard looks like an alternative that’s either partially or fully opensource? Not sure if there are any other good options

2 Likes
9

Thank you for taking this on; I feel it will greatly benefit the nix ecosystem.

I’m honing in on this goal as it resonates deeply with me.

Nix is excellent and I love what it provides. Yet I still struggle learning and teaching it.

The state of the community reminds me of the ember.js ecosystem somewhere between 2016 and 2018. They ended up forming a learning team which, in my opinion, had an incredibly successful impact on the community. They did that through high quality, official technical documentation and other community efforts.

Just my bystander reactions!

1 Like
10

I would be interested in participating in a technical writing team effort.

4 Likes
11

If such an effort starts around nix, announce it in #dev too :slight_smile:

1 Like