NixCon 2020 status update

[andir]

So… COVID-19 happenend.

We were planning to host this year’s NixCon in the first or second weekend of October, in Darmstadt, Germany. We had already quite final plans for a venue, recordings, scheduling and catering. The location is a University which has a lot of room for various hacking rooms. We were quite excited of how things would turn out.

Now because of the epidemic, we are not allowed to host events with more than 10 people until at least the end of September. Nobody knows what will be allowed afterwards, and whether there will be travel restrictions to people from other regions. 2020 seems to be a dud for physical meets.

We still want to organize something. NixCon 2020 Online Edition. This is a new format and we don’t know exactly how it will look like yet. If you are interested in helping out re-create some of the experience that we have with NixCon, but online, join us on the #nixcon channel to discuss.

Stay safe and healthy,
@megfault, @flokli, @andir, @zimbatm

[snajpa]

I feel like this is the perfect time and place to share this:

We (the guys from vpsFree.cz, mostly @martyet and me) have jumped on the train of organizing NixCon 2019 in Brno with some vision of continuity - that if we did that one year well, we could perhaps repeat that every year (or every N years and mix it with all the different locations, where the Nix community ends up finding capable people, who’d be on board with organizing the events locally long-term) .

Clearly, we didn’t communicate this well enough - though when it comes to budget, I’ve originally thought I’ve made it clear enough, that we’d try to save up what we could to enable even better NixCon in Brno sometime in (what we hoped to be) a near future.

So, after factoring in expenses (also factoring in a bit of compensation for all the full-time work at the time that @martyet did, we’ve ended up with net+ of ~5k8 EUR.

But then COVID-19 came and it turned things upside down here in CZ with a sudden and total lockdown (after a good few weeks of govt neglecting the whole situation, needless to say) caused a total disruption in communication within different communities (and what I learned later, as I don’t have kids yet, schools are probably the least prepared for stay-at-home situation like this).

So trying while trying to figure out, how we could help best I came along Zoom and I’ve deduced from the state of things, that this was going to be a huge mess and that we couldn’t rely on anything pretty much anyone from “the outside” says (at least for a while), so in the state of nuclear panic we deployed Jitsi Meet at meet.vpsfree.cz and made it available for anyone without the need for an account - just as the original upstream instance is, but that one was heavily overloaded at the time. Plus there is always the risk of loosing transit connectivity country-wide and what then…

So we did that and it quickly ended up eating up all the free CPU resources, we had - we had to find a way to borrow machines from somewhere at time, when everyone else was doing the same. So, we’ve decided we’d use the NixCon buffer to get us some additional cores.

Now we’re at 108 cores and the instance should be capable of handling NixCon, I think, pretty well - if we enable a few perks (which we didn’t like at the time, like recording and sending the stream to YT). 40 cores of those can be considered as belonging to the Nix community, if the community wants them

The situation being globally what it is, I hope noone will see this as a bad decision/route to take - more than anything, it now gives us an opportunity here to not stagnate, when it comes to social contact

So I’d like ask the community:

are we going to use the horsepower (and the left byuing power) for on-line socialization (and perhaps more collaborative on-line development environment) in the Nix community?

Is this something, where you guys would see us cooperating together longer-term?

Otherwise, the fair default is to, of course, re-pay the debt back to the NixCon/vpsFree account and then leave the decision-making process, about what to do with the funds, for later point of time, when the dust from this pandemic settles. That would mean settling a 3k6 EUR of the debt against our promise to leave any potential gains to the community.

(the account is transparent and the log is visible here: Internetové bankovnictví | Fio banka)

Let’s have a productive discussion about these things

/snajpa

[davidak]

I have often heard that BigBlueButton is more stable and scalable than Jitsi Meet.

The Flatten The Curve Summit switched shortly before the event to BBB from Zoom. https://flattenthecurve.tech/technical-support/

LibrePlanet 2020 from the FSF used Jitsi.

tl:dr
https://www.fsf.org/blogs/community/how-to-livestream-a-conference-in-just-under-a-week
https://www.fsf.org/blogs/community/looking-back-at-libreplanet-2020-freeing-the-future-together

The german Hostsharing cooperative tested BBB successfully with 200 participants! Zoom Alternative BigBlueButton™ im Praxistest | Hostsharing eG – die Hosting-Genossenschaft

The DiVOC also used BBB and Jitsi. No need for Zoom!

That’s everything i can contribute. I see forward to a very relaxed conference from home.

[snajpa]

BBB doesn’t scale with their open-source edition, you need their proprietary balancer - they don’t document any way to balance BBB with open-source means

Otherwise that was the best shot, when it comes to e-tutoring online FOSS…

Also I’d like to point out that we’re working with mmilata converting the Meet setup to Nix (it’s a quickly hacked together ugly setup with Ubuntu, currently; though I guess @davidak you’ve alread seen that one, for others, the current core implementation can be found here: GitHub - otevrenamesta/nur-packages: Our NUR repository, there’s an open PR with a version of this somewhere I guess).

During the peaks, we’ve got 1k participants so far - that’s way beyond any machine that we could find (with Meet, there’s approximately 5% of a CPU core consumed by a single participant on the server side, no matter which conference the participant is in).

By now, I have invested a lot more time in this, than I originally expected - I’d like to make that knowledge useful going forward, as I said above, we even have the HW means (and there’s one setup running live right now at https://meet.vpsfree.cz; but it’s in Czech by default).

It’s hosted in vpsAdminOS (https://vpsadminos.org) based containers, so we can easily host a few instances along that one (as long as the peak hours of those instances don’t match, that is).

[7c6f434c]

BBB doesn’t scale with their open-source edition, you need their proprietary balancer - they don’t document any way to balance BBB with open-source means

On the one hand, I am not even sure you can scale a single BBB room with the proprietary balancer — one just needs a large enough server, apparently.

And I guess balancing is critical for the 100×20 scenario, for two parallel tracks each needing most of a server once could with the same effort setup two servers and post room URLs leading to different servers.

On the other hand, there seems to be scalelite?

During the peaks, we’ve got 1k participants so far - that’s way beyond any machine that we could find (with Meet, there’s approximately 5% of a CPU core consumed by a single participant on the server side, no matter which conference the participant is in).

Does this depend on the share of participants with the cameras on?

[davidak]

@snajpa

BBB doesn’t scale with their open-source edition, you need their proprietary balancer

oh, very sad to hear that

@snajpa

there’s an open PR with a version of this somewhere I guess

yes, i also follow that https://github.com/NixOS/nixpkgs/pull/82920

Jitsi Meet is a good software which i use for years now, but it has minor issues now and then

@snajpa

During the peaks, we’ve got 1k participants so far

that’s probably also more people than wanting to attend NixCon? and do they have to be in the room all the time? the lectures can just be streamed to youtube or maybe https://openstreamingplatform.com/ or some other setup (simple Nginx-RTMP). for discussions, Q&A, hackathon etc are small rooms good enough, they could even be on different servers that are just rented for the event (Cloud VPS). but you probably thought about all that…

(i haven’t read your long post completely, so i might miss some detail about hardware. i don’t distract you further )

[domenkozar]

@snajpa

Is this something, where you guys would see us cooperating together longer-term?

Otherwise, the fair default is to, of course, re-pay the debt back to the NixCon/vpsFree account and then leave the decision-making process, about what to do with the funds, for later point of time, when the dust from this pandemic settles. That would mean settling a 3k6 EUR of the debt against our promise to leave any potential gains to the community.

If the promise was made that the conference surplus goes back to the community (and I assume further nixcons), then that would be the only sane way to budget this.

If not, then it’s up to vpsFree to judge best to do with their money.

Zoom business plan costs 19 EUR / month for up to 300 participants which would be much cheaper option for the community than DIY hosting with the bought equipment.

[snajpa]

Yes, the promise was made by me, assuming a few things about the community, which weren’t entirely based on reality.

1. I was assuming, that we’d have that budget avail for another conf in Brno, which is something we can do, but we surely don’t have the spare time to organize anything anywhere else, sadly (we’re a non-profit community too)

2. I was assuming right from the beginning, that there was a functional Foundation and that we wouldn’t have to mess around with accounting that stuff - and that would save time & headaches & any gains would stay right at the Foundation

3. Domen, from what you’re writing, I’m pretty much disappointed even more.

I thought that the Nix community is made of people who understand the value of privacy, I certainly didn’t expect an argument in EUR for Zoom so soon.

So, in conclusion, I’m inclined to do these things:

1. keep my promise to the Nix community and send the $ (when we’re able to, which is up for discussion) to the project, somewhere, which I have no idea where, when the Foundation is ± dead…

2. due to the state of the things, I don’t think we wish to be involved in the Nix community all that much, the excitement is just not there anymore - especially after the summarizing meeting after last NixCon in Brno, where everyone was super-keen to point out even the slightest mistakes we made that year - but look, where are we now. There was a #1 thing on the list of things of ‘what to do better next year’, and that was ‘start the hell earlier’ - remind me, where were we with NixCon before COVID-19 hit?

So that’s that, looks like a smallish chapter in one’s life is about to close, so I’m excited what comes next

[rbvermaa]

@snajpa

… especially after the summarizing meeting after last NixCon in Brno, where everyone was super-keen to point out even the slightest mistakes we made that year …

Sorry to hear. For what it is worth, I’ve heard many people say only positive words about NixCon 2019 and the organization. Personally, I thought the event was great.

[danieldk]

@snajpa

I thought that the Nix community is made of people who understand the value of privacy, I certainly didn’t expect an argument in EUR for Zoom so soon.

On Zoom specifically: they are evil and all. But much of it can be mitigated by using Zoom in a web browser. I attended a conference via Zoom (the Oxidize 1K embedded Rust conference) and at least for me as an attendant it was flawless.

Generally speaking, if the price deltas are very large, it can be good to be pragmatic about these things. 19 Euro for hosting conference video feeds leaves a lot of money on the table for other things.

[domenkozar]

@snajpa

I was assuming, that we’d have that budget avail for another conf in Brno, which is something we can do, but we surely don’t have the spare time to organize anything anywhere else, sadly (we’re a non-profit community too)

I’m not sure where the miscommunication happened nor what was communicated, but so far the conference was at a different location each year. That doesn’t mean it can’t be at the same place twice (since there’s overhead), but there are also international members helping the organization as far as I know, so again there must have been a miscommunication.

@snajpa

I thought that the Nix community is made of people who understand the value of privacy, I certainly didn’t expect an argument in EUR for Zoom so soon.

I value privacy a lot, but it’s not clear to me if an event that has always been publicly streamed has any privacy to be considered?

@snajpa

especially after the summarizing meeting after last NixCon in Brno, where everyone was super-keen to point out even the slightest mistakes we made that year - but look, where are we now.

That’s a very common culture in developer world to point out how to improve, as we are problem solvers. It’s so easy to dismiss hours of hard work with a snarky comment (even easier on the internet).

I know that I can say that having built Cachix for two years that it’s something you have to get used to or you quit.

Nix community has grown quite a bit in recent years so we get a mixture of cultures from all over the development world, but we can do better by raising awareness how much work it is to organize a conference (especially for no financial reward!).

[snajpa]

Fair enough, there are a lots of mistakes on all sides involved

So, how do we learn from them best and what’s the next best course of action?

To the point of the conf being public - yes, of course, but I’ve mixed two things together here - and that’s not exactly fair to keep doing that, so let’s distinguish the two:

1. there is the conference itself, which given the fact it will be streamed, won’t use up that much resources anyway, so it doesn’t need that big of a cluster

2. I thought a general platform for private meetings of the community members could have some value (and that’s where the privacy concerns over Zoom & such kick in primarily)

Overall, there are more possibilities, what to do with the HW, than just running Jitsi Meet on there.

We’ve discussed that privately with @zimbatm as an early stage brainstorming; I’ve only contacted him after we really needed to panic-buy some HW for the Meet instance (due to the schools using it for online education) - there wasn’t enough time to ask for permission first, for which I’m sorry, but as I said - I do consider it being a debt against our initial promise. We should be able to pay that back in a next few following months, it was just really a better idea to do this, than to borrow money from the outside (at not-that-nice-for-a-non-profit interest rates).

The idea we’ve had is that not only we can run Jitsi on the two machines, we could provide the standard environment of our vpsAdminOS virtual servers (containers designed to behave as much as standalone VM), to the Nix community - for the most active developers, so that they could build & test stuff somewhere.

We only went after CPU cores per price ratio, when ordering the HW, so it’s got 20 cores each + HT (2 servers, total 40 cores) - but it doesn’t have a whole lot of RAM and storage. If we equip the nodes up to a standard level of what we do for standard host machines, each of those machines could be used by up to 80 people, making that 160 containers of our standard size (aggregated access to 8 CPU threads, 4G of user-space memory, 120G of disk space), which could be used by Nix community.

As I’ve told @zimbatm, we would like to do something for the Nix community long-term, so we’d host that hardware (and sponsor the electricity + connectivity), we could help with on-line NixCon, but I don’t see us much doing a next conference, as the setup behind organizing those is a mess and I’m not in a position to really change that

[FRidh]

I think hosting an online version is a great idea given the circumstances.

Considering there are still so many unknowns, differences between countries, and likely the big desire/need for people to do other things as soon as travel is possible, I suggest to push this online NixCon a bit forward, e.g. beginning of September. If possible, of course.

[armijn]

@snajpa

due to the state of the things, I don’t think we wish to be involved in the Nix community all that much, the excitement is just not there anymore - especially after the summarizing meeting after last NixCon in Brno, where everyone was super-keen to point out even the slightest mistakes we made that year

I helped organized (international) tech conferences for well over ten years and there are always things that can be improved. As an organizer you become oversensitive to these things (I have been there many times) and things can get blown out of proportion. Personally I thought that NixCon 2019 was very well run.

[tbenst]

I vote zoom. Presentation mode works super well, and allows for moderator control of who can speak, manage Q&A, etc. there’s been several thesis defenses and academic conferences on zoom and it’s worked great. Plus it seems to have the highest quality video of any of the services I’ve used, and many in the community probably already have a corporate account so could be hosted at 0 cost.

No one seems to be saying it so maybe I’m missing something, but spending surplus on privately used instance is highly questionable at first blush. Maybe people at Nix foundation / Nix community were consulted first. Were those 1k people connected concurrently associated with Nix? If not it sounds like a misuse of funds.

I’m not familiar with Nix Foundation / how NixCon is organized or financed, but seems like money should remain in the nonprofits bank account & that a nonprofit should bear financial risk of a conference.

[gilligan]

I neither took part in organizing NixCon 2019 nor did i attend. I only acted as facilitator for NixCon 2018 in London and have some experience in organizing conferences.

From this “outside perspective” (if you will):

it doesn’t come as a shock to me that there is going to be friction when 5000 € are being spent without (as it seems) transparent, public discussion.

I can think of a lot of ways to invest money for either an upcoming NixCon or the Nix community in general. Purchasing hardware and considering how to make this useful to either NixCon or the community in general as an /afterthought/ can only be described as unfortunate. Note that i am not saying my ideas would have been the right use of the money. More like it is far from obvious what the best use might be so this should have been evaluated i guess.

So if people are disgruntled about that, i think that cannot be dismissed.

I guess the way forward is to be explicit about how to deal with surplus in the future so situations like this can be avoided.

EDIT none of this should in any way belittle the energy, time, passion and effort spent by all the organizer of NixCon 2019. I was happy to be able to watch from afar and i appreciate that.

[snajpa]

I’m not sure, but I do belive it is visible, from what I’m writing and all of these points were already made, but I’ll summarize them once more:

1. It isn’t a standard situation that the margin calculacted to cover the potential loss, is at all sent to the community, as I announced originally we would do - as far as I can tell, the ‘surplus’ from previous Nixcon is still at the balance of original organizer. I’m not saying this is either good or bad, if anything, it is more evidence that the process to organize NixCons is broken;

2. If you compare the value we’ve got for the money in London to Brno… I don’t know, I thought that was pretty obvious and I didn’t have to point that out - but also due to all the things we decided to do extra (like the party on premises), we needed to leave much bigger headroom in the budget for mistakes and problems (broken stuff, etc.).

3. COVID-19 was an unexpected hit for all of us and sitting on this ‘surplus’ from Nixcon, we decided to do even more community work (not everything is only about Nix, you know?) - and enable at least some schools at least some function through online meet platform at a time, where none of the big guys stepped up to do so. Keeping them reliant on a thing with links to the Chinese? Pfff.

4. Like I’ve said the first time, I am offering to do even more community work for the Nix community (hosting dev environments for developers isn’t anything and of any use to anyone?) and I have tried apologizing for not consulting anyone before using a part of the leftover Nixcon Brno surplus to cover our need for HW for Meet.

5. Also, I’ve said that we do intend to re-pay that back, where I tried to put it in a way that would be understood, that I see that as - I don’t know - maybe a potential loss to the community, when we could cooperate further and make something more of it?

I’m not expecting standing ovations or anything remotely closely to it, I was just expecting more understanding approach from the community.

If that’s not possible, just send me the IBAN and let’s be done with it.

[snajpa]

Should anyone find interest in that, the timeline with Meet went something like this:

03/16 a total lockdown instituted in CZ (ping me on irc if you want further comments on that )
03/17 we’ve launched meet.vpsfree.cz and announced it, it was running on a machine with 20 cores at the time, which we thought, OMG that should be enough, … or?
03/18 well, it’s not going to be nearly enough
03/19 still replying to teacher’s e-mails, that we would do everything we could to help them teach
03/20 panicking, seeing that we can’t cover that by any means on our own, well shit, we’re a non-profit and can’t easily access any kind of short-term financing, but we’re a community, so we’re trying to borrow HW everywhere we can
03/25 panicking even more, finding out guys at local universities would love to help, but only have environments with VMs that can’t handle as much UDP traffic as we need…
03/28 we’re at max of racking any HW we could get (+72 cores, at a time where everyone else was too looking for some extra horsepower, not that bad)
03/30 well shit, even that’s not going to be enough
03/31 we panic-ordered +40 cores of additional HW, just barely configured to get us as many cores as we can get

Then it was a fight still until Skype announced that they too, would do videoconferencing without registration and via a web browser, so with no setup overhead. And then everyone else, then the orders to use Teams kicked in in companies, so that lowered the use even more.

Now some of the users are returning as the various other platforms keep having problems, but it’s not nearly as hot as is was a few weeks ago.

I thought I might regret taking this path, but I hoped it won’t come to it - I chose to believe in the better in people.

^^ community powa! The guy who was nice enough to lend us that HW even racked it at a time, when we weren’t able to get to the DC The pizzabox on the side doesn’t have a cover, it’s an older engineering sample of dual node OpenCompute system - that’s how desperate we were - so it’s turned on a side (we couldn’t put the paper boxes on and didn’t find a shelf in time…)

[danieldk]

@tohl2

everybody else excluded?

Zoom can be used in a browser.

[caaaaaa]

My experiences with using zoom on nixos quite a lot for university lectures this last weeks:

In theory the web interface runs on chromium and works, but it is rather unstable. It seems that if there is package loss, the audio stream gets instantly dropped, and i had to reload the site, sometimes every five minutes. (I cant do much about my connection here right now) Other participants reported the same behavior. Zoom also does a lot to push the client, hiding the webinterface behind countdowns. (University also mentioned that the webinterface is not supposed to be used)

So for me there was no alternative to install this random unfree binary from nixpkgs, that works well and handles bad connections. But i still have bad feelings about it, see for example this article from last year: https://medium.com/bugbountywriteup/zoom-zero-day-4-million-webcams-maybe-an-rce-just-get-them-to-visit-your-website-ac75c83f4ef5

If there is an offer for hosting of free alternatives, i would definitely take it, even if it should cost a bit more.