Firefox-sync 401 login failed due to misconfigured nginx

fschn90 · March 8, 2025, 12:30pm

For the love of god i just dont get it to work. I followed the nix manual instructions. But I keep getting 401 login errors:

Sync.Resource	DEBUG	GET fail 401 https://firefox-sync.fschn.org/1.5/4/info/collections
Sync.Resource	WARN	GET request to https://firefox-sync.fschn.org/1.5/4/info/collections failed with status 401
Sync.Service	WARN	401: login failed.

more of the firefox about:sync-log are blow. im really out of ideas now. any leads and help is highly appreciated.

my basic setup is:

{ pkgs, config, ... }:
{
  services.mysql.package = pkgs.mariadb;
  services.firefox-syncserver = {
    enable = true;
    secrets = config.sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET".path;
    singleNode = {
      enable = true;
      hostname = "firefox-sync.fschn.org";
      enableNginx = true;
    };
  };
  sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET" = { 
    mode = "0777";
  };
  services.nginx = {
    virtualHosts."firefox-sync.fschn.org" = {
      useACMEHost = "fschn.org";
      forceSSL = lib.mkForce true;
    };
  };
}

the secret is set with:

SYNC_MASTER_SECRET=$(cat /dev/random | base32 | head -c64)

testing if the server is up:

$ curl https://firefox-sync.fschn.org/__heartbeat__ 
{"database":"Ok","quota":{"enabled":false,"size":0},"version":"0.17.12","status":"Ok"}

Firefox Browser configuration process:

Sign out of mozilla account in firefox sync settings
Close Firefox Browser
cd to firefox profile dir & mv weaver weaver-bak #firefox-sync used to be called weaver
Open Firefox Browser
Navigate to about:config in your Firefox address bar and set identity.sync.tokenserver.uri to https://firefox-sync.fschn.org/1.0/sync/1.5
Sign in moxilla account

My setup of machine running firefox-sync:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.74, NixOS, 24.11 (Vicuna), 24.11.20250225.5d7db46`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.12`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/store/lb6ypkpf38qsd0p4gc5nqvb97s2brh5h-source`

Setup of machine running firefox browser to sync:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.12.16, NixOS, 24.11 (Vicuna), 24.11.20250225.5d7db46`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.12`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/store/lb6ypkpf38qsd0p4gc5nqvb97s2brh5h-source`
$ firefox --version                                 
Mozilla Firefox 135.0.1

Whole Firefox about:sync-log:

1741436354189	Sync.LogManager	DEBUG	Flushing file log
1741436354195	Sync.LogManager	DEBUG	Log cleanup threshold time: 1740572354195
1741436354195	Sync.LogManager	DEBUG	Done deleting files.
1741436357291	FirefoxAccounts	INFO	fetching updated device list
1741436357303	Services.Common.RESTRequest	DEBUG	GET request to https://api.accounts.firefox.com/v1/account/devices?filterIdleDevicesTimestamp=1739621957291
1741436357534	Services.Common.RESTRequest	DEBUG	GET https://api.accounts.firefox.com/v1/account/devices?filterIdleDevicesTimestamp=1739621957291 200
1741436357534	Hawk	DEBUG	(Response) /account/devices?filterIdleDevicesTimestamp=1739621957291: code: 200 - Status text: 
1741436357534	Hawk	DEBUG	Clock offset vs https://api.accounts.firefox.com/v1: -534
1741436357534	FirefoxAccounts	INFO	Got new device list: 1c484495fc2fc42abfb6ce7241781078, 87bb311d18f51c4e695ff5d42ca68efe, 9b4ffac506b296deae62bf7b090f3eca
1741436357534	FirefoxAccounts	TRACE	Getting command keys: https://identity.mozilla.com/cmd/open-uri
1741436357534	FirefoxAccounts	TRACE	Getting command keys: https://identity.mozilla.com/cmd/close-uri/v1
1741436357534	FirefoxAccounts	TRACE	Our push subscription looks OK
1741436357534	FirefoxAccounts	INFO	updating the cache
1741436365848	FirefoxAccounts	INFO	Polling device commands.
1741436365848	Sync.Service	DEBUG	User-Agent: Firefox/135.0.1 (Linux x86_64) FxSync/1.137.0.20250216192613.desktop
1741436365848	Sync.Service	INFO	Starting sync at 2025-03-08 13:19:25 in browser session Xe2RkkEZL1q7
1741436365848	Sync.Service	DEBUG	In sync: should login.
1741436365848	Sync.Service	INFO	User logged in successfully - verifying login.
1741436365848	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1741436365848	Sync.SyncAuthManager	DEBUG	unlockAndVerifyAuthState already has (or can fetch) sync keys
1741436365848	Sync.Status	DEBUG	Status.login: error.login.reason.network => success.status_ok
1741436365848	Sync.Status	DEBUG	Status.service: error.login.failed => error.login.failed
1741436365848	Sync.Service	DEBUG	Fetching unlocked auth state returned success.status_ok
1741436365849	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1741436365850	Services.Common.RESTRequest	DEBUG	GET request to https://api.accounts.firefox.com/v1/account/device/commands?index=0
1741436365862	Sync.Resource	DEBUG	GET fail 401 https://firefox-sync.fschn.org/1.5/4/info/collections
1741436365862	Sync.Resource	WARN	GET request to https://firefox-sync.fschn.org/1.5/4/info/collections failed with status 401
1741436365862	Sync.Service	WARN	401: login failed.
1741436365862	Sync.SyncAuthManager	DEBUG	_findCluster has a pre-existing clusterURL, so fetching a new token token
1741436365863	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1741436365863	Sync.SyncAuthManager	INFO	Getting sync key
1741436365863	FirefoxAccounts	DEBUG	getOAuthToken enter
1741436365863	FirefoxAccounts	TRACE	getCachedToken returning cached token
1741436365863	FirefoxAccounts	DEBUG	getOAuthToken returning a cached token
1741436365863	Sync.SyncAuthManager	INFO	Getting a sync token from: https://firefox-sync.fschn.org/1.0/sync/1.5
1741436365863	Sync.SyncAuthManager	DEBUG	Getting a token using OAuth
1741436365863	Services.Common.TokenServerClient	DEBUG	Beginning OAuth token exchange: https://firefox-sync.fschn.org/1.0/sync/1.5
1741436365863	Services.Common.RESTRequest	DEBUG	GET request to https://firefox-sync.fschn.org/1.0/sync/1.5
1741436365888	Services.Common.RESTRequest	DEBUG	GET https://firefox-sync.fschn.org/1.0/sync/1.5 200
1741436365888	Services.Common.TokenServerClient	DEBUG	Got token response: 200
1741436365888	Services.Common.TokenServerClient	DEBUG	Successful token response
1741436365888	Sync.Status	DEBUG	Status.login: success.status_ok => success.login
1741436365888	Sync.Status	DEBUG	Status.service: error.login.failed => success.status_ok
1741436365888	Sync.SyncAuthManager	DEBUG	_findCluster returning http://firefox-sync.fschn.org/1.5/4/
1741436365888	Sync.SyncAuthManager	DEBUG	Cluster value = http://firefox-sync.fschn.org/1.5/4/
1741436365888	Sync.Status	DEBUG	Status.login: success.login => error.login.reason.network
1741436365888	Sync.Status	DEBUG	Status.service: success.status_ok => error.login.failed
1741436365888	Sync.ErrorHandler	ERROR	Sync encountered a login error
1741436365889	Sync.SyncScheduler	DEBUG	Clearing sync triggers and the global score.
1741436365889	Sync.SyncScheduler	DEBUG	Next sync in 3600000 ms. (why=schedule)
1741436365889	Sync.Service	DEBUG	Exception calling WrappedLock: Error: Login failed: error.login.reason.network(resource://services-sync/service.sys.mjs:1041:15) JS Stack trace: onNotify@service.sys.mjs:1041:15
1741436365889	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1741436365889	Sync.Service	DEBUG	Not syncing: login returned false.
1741436365889	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent

Dirakon · March 16, 2025, 4:43pm

Also tried to make it work, getting the same 401 on collections. Trying to figure it out still

charludo · March 20, 2025, 3:32pm

Exact same issue here. I “resolved” it by syncing with the official Mozilla Servers, then logging out of Sync, but not deleting the associated data when prompted.

Afterwards, change the Sync URL to your own server. Then create a NEW Mozilla account, and sing in with that one. Now sync works.

Stupid, I know…

charludo · March 20, 2025, 5:00pm

Update: aaaaaand it’s gone again.

Edit: This seems to have solved it (for now at least…): Syncserver not working with error .../1.5/1/info/collections failed with status 401 in sync-log · Issue #1564 · mozilla-services/syncstorage-rs · GitHub

I had a type in the name of the secret

fschn90 · April 15, 2025, 1:25pm

Is it still working @charludo ?

Do you mind sharing how your specify services.firefox-syncserver.secrets with config.age.secrets.ffsync.path?

i specify my secret with SYNC_MASTER_SECRET=$(cat /dev/random | base32 | head -c64) which gives me a sample output of:

SYNC_MASTER_SECRET=ACIOBUPFOE4C3SIBL4MWDVJMEL6S6YRJIATT3DECS4VPCI4YN4XPFVZXPWPUJZFW

from your answer i suppose my error is here somewhere.

really appreciate your help

charludo · April 15, 2025, 1:41pm

Yes, it’s been working flawlessly! My config looks like this:

  age.secrets.ffsync.rekeyFile = secrets.ffsync;
  services.firefox-syncserver = {
    enable = true;
    secrets = config.age.secrets.ffsync.path;
    settings.host = "0.0.0.0";

    singleNode = {
      enable = true;
      capacity = 2;
      hostname = "0.0.0.0";
      url = "https://ffsync.${private-settings.domains.home}";
    };
  };

(plus opening the ports)

Note that you do not pas in the actual secret (no using cat or the like), you pass in the path were the secrets file can be found. In my case, that is managed through agenix, but you could in theory just place a file wherever manually and point FFSync to it (don’t do that though. Use some form of proper secrets management).

Also, I am pretty sure that secret should remain static, not be changed upon every rebuild (though correct me if I am wrong )

But yes, the contents of that rekey file look like

SYNC_MASTER_SECRET=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

fschn90 · April 15, 2025, 3:07pm

wow, thank you for your reply @charludo ! i tried opening the ports now (still not working) and a did have the secret specified statically via sops-nix but still getting these errors tho …

# firefox about:sync-log
1744728763810	Sync.Service	DEBUG	User-Agent: Firefox/136.0.2 (Linux x86_64) FxSync/1.138.0.20250317120031.desktop
1744728763810	Sync.Service	INFO	Starting sync at 2025-04-15 16:52:43 in browser session YuqJBbW_uLMh
1744728763810	Sync.Service	DEBUG	In sync: should login.
1744728763810	Sync.Service	INFO	User logged in successfully - verifying login.
1744728763810	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1744728763810	Sync.SyncAuthManager	DEBUG	unlockAndVerifyAuthState already has (or can fetch) sync keys
1744728763810	Sync.Status	DEBUG	Status.login: error.login.reason.network => success.status_ok
1744728763810	Sync.Status	DEBUG	Status.service: error.login.failed => error.login.failed
1744728763810	Sync.Service	DEBUG	Fetching unlocked auth state returned success.status_ok
1744728763810	FirefoxAccounts	TRACE	not checking freshness of profile as it remains recent
1744728763823	Sync.Resource	DEBUG	GET fail 401 https://firefox-sync.fschn.org/1.5/4/info/collections
1744728763823	Sync.Resource	WARN	GET request to https://firefox-sync.fschn.org/1.5/4/info/collections failed with status 401
1744728763823	Sync.Service	WARN	401: login failed.

ill try copying your config now, hope thats okay

charludo · April 15, 2025, 3:25pm

Out of curiosity, what URl do you put into identity.sync.tokenserver.uri in FIrefox?

fschn90 · April 15, 2025, 3:41pm

https://firefox-sync.fschn.org/1.0/sync/1.5

based on my config, still same as above

{ pkgs, config, ... }:
{
  services.mysql.package = pkgs.mariadb;
  services.firefox-syncserver = {
    enable = true;
    secrets = config.sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET".path;
    singleNode = {
      enable = true;
      hostname = "firefox-sync.fschn.org";
      enableNginx = true;
    };
  };
  sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET" = { 
    mode = "0777";
  };
  services.nginx = {
    virtualHosts."firefox-sync.fschn.org" = {
      useACMEHost = "fschn.org";
      forceSSL = lib.mkForce true;
    };
  };

fschn90 · April 17, 2025, 3:29pm

i’m really sorry to keep bothering you with questions @charludo. im trying to copy your config, but i dont understand it.

Do you have a reverse proxy for services.firefox-syncserver.singleNode.url? I dont find anywhere one in the rest of your config Where and how do you manage that url?
What do you put in as identity.sync.tokenserver.uri? does it work out to https://ffsync.${private-settings.domains.home}/1.0/sync/1.5?

really appreciate your help. but if i have overstretched it already, then im sorry

charludo · April 17, 2025, 5:44pm

No worries!

Yes, I have a reverse proxy for that URL. Sorry, I forgot to mention that. In my case, it’s running directly on my firewall, but you should be able to just use (e.g.) an nginx config directly on the same machine.

You can find my config here: nix/vms/SRV-FFSYNC.nix at 6b1ef6dbfcf9d306903fdaad47210bcd14da51dd · charludo/nix · GitHub

For FF, my config contains this:

"identity.sync.tokenserver.uri" = "https://ffsync.${private-settings.domains.home}/1.0/sync/1.5";

so yes, exactly what you wrote.

fschn90 · April 22, 2025, 11:14am

it now works, at last! i think the problem was misconfiguring nginx:

  services.firefox-syncserver.singleNode.enableNginx = true;
  # at the same time trying to get the TLS working with:
  services.nginx = {
     virtualHosts."firefox-sync.fschn.org" = {
       useACMEHost = "fschn.org";
       # avoiding error `The option has conflicting definition values` with lib.mkForce
       forceSSL = lib.mkForce true;
     };
   };

enabling nginx with a firefox-syncserver option and at the same time configuring options via nginx like above seems to have caused the malfunction.

keeping firefox-syncserver and nginx separated and clean like this blow works like a charm:

{ lib, pkgs, config, ... }:

{
  services.mysql.package = pkgs.mariadb;
  sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET" = { };
  services.firefox-syncserver = {
    enable = true;
    secrets = config.sops.secrets."firefox-syncserver/SYNC_MASTER_SECRET".path;
    settings.host = "0.0.0.0";
    singleNode = {
      enable = true;
      capacity = 4;
      hostname = "0.0.0.0";
    };
  };

  services.nginx = {
    virtualHosts."ffsync.fschn.org" = {
      useACMEHost = "fschn.org";
      forceSSL = true;
      locations."/" = {
        proxyPass = "http://localhost:5000";
      };
    };
  };
}

thank you so much @charludo for your support!

charludo · April 22, 2025, 11:25am

Nice, great to see that it’s working for you!