Is there a feasible way to debug/test hydra darwin failures quickly?

Development
1

While contributing to ZHF I experienced some general issues that some python packages fail only on hydra x86_64_darwin but work on x86_64_linux:

In this particular case hydra seems to have problems with the following test only on darwin:

fs = <fsspec.implementations.local.LocalFileSystem object at 0x10c00f190>
temp_file = '/private/tmp/nix-build-python3.8-fsspec-0.7.4.drv-0/tmpsupgb68itest-file'

    @pytest.mark.parametrize("fs", ["local"], indirect=["fs"])
    def test_modified(fs: AbstractFileSystem, temp_file):
        try:
            fs.touch(temp_file)
            created = fs.created(path=temp_file)
            time.sleep(0.05)
            fs.touch(temp_file)
            modified = fs.modified(path=temp_file)
            assert isinstance(modified, datetime.datetime)
>           assert modified > created
E           assert datetime.datetime(2020, 9, 10, 19, 8, 22) > datetime.datetime(2020, 9, 10, 19, 8, 22)

fsspec/implementations/tests/test_common.py:29: AssertionError

Strange. Why does it have problems comparing timestamps? Anyway next step is to test it locally. Unfortunately all the tests pass on my machine (OSX 10.15).

So what to do now? It works locally on darwin and on hydra linux but fails with a only-reproducible error on hydra darwin.

In this case the usual approach is to just disable the one failing test on darwin so that hydra darwin passes. With the new pytestCheckHook this can be done in a pretty elegant way as can be seen in this accepted pull request.

I then backported this change into 20.09 only to see that now another test case fails on hydra darwin only. (a new test case because I also bumped the version which had additional tests).

So now the game begins again:

  1. disable test on darwin
  2. wait until reviewed
  3. see if it builds on hydra

So long story short: Can I somehow test darwin builds on hydra locally to get faster feedback? Is there a possibility to optimise this workflow? And finally: Can somebody explain why the test_modified actually failed on hydra? :grinning:

1 Like
2

It looks to me like the test slept for 50ms and hoped that the timestamp
would change. However the printing of the timestamp only shows second
granularity. Possibly the filesystem that it is running on doesn’t
support that timestamp resolution? Possibly the builder that this test
ran on and your local machine use different filesystems. According to
unreliable sources HFS+ has second timestamp resolution while APFS has
nanosecond resolution.

If you see a test with sleep in it you know you are going to have a bad
time :stuck_out_tongue:

3

Possibly the builder that this test
ran on and your local machine use different filesystems. According to
unreliable sources HFS+ has second timestamp resolution while APFS has
nanosecond resolution.

Ah interesting point. Thanks for sharing. On which Mac OS is the darwin builder running? Can we change the filesystem of the builder to APFS as well?

If you see a test with sleep in it you know you are going to have a bad
time :stuck_out_tongue:

So what do you do with these tests? Pray that the pass and if not disable them :smiley:?

4

| tricktron
September 29 |

  • | - |

Possibly the builder that this test
ran on and your local machine use different filesystems. According to
unreliable sources HFS+ has second timestamp resolution while APFS has
nanosecond resolution.

Ah interesting point. Thanks for sharing. On which Mac OS is the darwin builder running? Can we change the filesystem of the builder to APFS as well?

I don’t really macOS so I have no idea. I also don’t know anything about our macOS Hydra setup.

So what do you do with these tests? Pray that the pass and if not disable them :smiley:?

Yeah, that sounds like the best approach for package maintainers. You can report this issue upstream and see what they thing. Unfortunately sometimes tests with time is necessary. In that case they can probably use touch instead of sleep which would actually give them a better test anyways and avoid testing the filesystem.