Skip failing FBF tests in Google Chrome when running makeref on the bots

[Snuffleupagus]

Note that this will still allow the FBF tests to run locally, and also on the bots when invoked with test/browsertest (to not lose all the FBF test-coverage), but will no longer prevent makeref from running successfully on the bots.

[Snuffleupagus]

/botio test

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.215.176.217:8877/17df7dae3782e5a/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.67.70.0:8877/79bbd19bbcbba54/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.215.176.217:8877/17df7dae3782e5a/output.txt

Total script time: 2.19 mins

• Font tests: FAILED
• Unit tests: FAILED
• Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/17df7dae3782e5a/reftest-analyzer.html#web=eq.log

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/79bbd19bbcbba54/output.txt

Total script time: 19.63 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/79bbd19bbcbba54/reftest-analyzer.html#web=eq.log

[Snuffleupagus]

Let's try this...
/botio makeref

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_makeref from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.215.176.217:8877/8b6450c2d07c38f/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_makeref from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.67.70.0:8877/41b76b9689b1ccc/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.215.176.217:8877/8b6450c2d07c38f/output.txt

Total script time: 2.53 mins

• Lint: Passed
• Make references: FAILED

[pdfjsbot]

From: Bot.io (Linux m4)

---

Success

Full output at http://54.67.70.0:8877/41b76b9689b1ccc/output.txt

Total script time: 18.10 mins

• Lint: Passed
• Make references: Passed
• Check references: Passed

[Snuffleupagus]

/botio test

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.67.70.0:8877/c86cc108caf9ee0/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @Snuffleupagus received. Current queue size: 0

Live output at: http://54.215.176.217:8877/031eebd916d6d4f/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.215.176.217:8877/031eebd916d6d4f/output.txt

Total script time: 2.12 mins

• Font tests: FAILED
• Unit tests: FAILED
• Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/031eebd916d6d4f/reftest-analyzer.html#web=eq.log

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/c86cc108caf9ee0/output.txt

Total script time: 19.57 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/c86cc108caf9ee0/reftest-analyzer.html#web=eq.log

[Snuffleupagus]

The Windows bot seem to need a restart, or something...

The Linux bot mostly works now, but unfortunately there's a number of Google Chrome test failures that seem intermittent. However, we might just have to live with this and simply disregard those failures when looking at the test logs!?

[brendandahl]

/botio-windows makeref

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/922d526d0b914ba/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Success

Full output at http://54.215.176.217:8877/922d526d0b914ba/output.txt

Total script time: 24.36 mins

• Lint: Passed
• Make references: Passed
• Check references: Passed

[brendandahl]

/botio-windows test

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/22bbfea3b02da45/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.215.176.217:8877/22bbfea3b02da45/output.txt

Total script time: 26.49 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/22bbfea3b02da45/reftest-analyzer.html#web=eq.log

[brendandahl]

It appears chrome on windows and linux is no longer reproducible. Last time we were able to extract the rendering that had an issue and file a bug with chromium (which I think they fixed).

[brendandahl]

/botio test

[pdfjsbot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.215.176.217:8877/83da448af34b58e/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/1ad1f70ae5804c2/output.txt

[pdfjsbot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.215.176.217:8877/83da448af34b58e/output.txt

Total script time: 26.21 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.215.176.217:8877/83da448af34b58e/reftest-analyzer.html#web=eq.log

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/1ad1f70ae5804c2/output.txt

Total script time: 60.00 mins

[brendandahl]

/botio-linux test

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/6c0f88ebf69f9cb/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/6c0f88ebf69f9cb/output.txt

Total script time: 14.80 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/6c0f88ebf69f9cb/reftest-analyzer.html#web=eq.log

[brendandahl]

/botio-linux makeref

Trying out headless mode to see if it's stable.

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/21a9df5ba9a3069/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/21a9df5ba9a3069/output.txt

Total script time: 29.30 mins

• Lint: Passed
• Make references: FAILED

[timvandermeij]

Since Google Chrome is so inconsistent now, should we just disable it entirely for now so we can at least continue development, and try to enable it later on again using a follow-up issue?

[Snuffleupagus]

Trying out headless mode to see if it's stable.

Unfortunately that didn't seem to help :-(
It seems that headless also made the tests slower, unless I'm missing something.

Since Google Chrome is so inconsistent now, should we just disable it entirely for now so we can at least continue development, and try to enable it later on again using a follow-up issue?

Either that, or we continue to run tests in Chrome and simply disregard these intermittent failures when looking at the results (although I suppose that this could increase the risk of actual regressions sneaking past a reviewer).

[timvandermeij]

I noticed that the Linux bot seems down right now since I cannot reach previews or reference image comparisons.

How about we do the the following?

1. Switch both both from headless mode back to non-headless mode since that didn't make a difference in stability and seemed to have make the bots a bit slower.
2. Disable Chrome on both bots for now and make a follow-up issue to determine the cause.
3. Restart both bots to start fresh.

We could replace step 2 with only disabling the FBF tests as done in this patch, but then we'll have to ignore the intermittent failures which might make it harder to see what actually failed. However, one could argue that we then at least run most of the tests in Chrome. I'm on the fence about this step, so what do you think @brendandahl?

[brendandahl]

I think we try to leave chrome chrome enabled for now and see how bad the intermittents are. If we're spending too much time validating tests then we can disable.

[brendandahl]

/botio-linux makeref

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_makeref from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/b56556d37a42104/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Success

Full output at http://54.67.70.0:8877/b56556d37a42104/output.txt

Total script time: 18.11 mins

• Lint: Passed
• Make references: Passed
• Check references: Passed

[brendandahl]

/botio-linux test

[pdfjsbot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @brendandahl received. Current queue size: 0

Live output at: http://54.67.70.0:8877/fde8c42f6858c2d/output.txt

[pdfjsbot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.67.70.0:8877/fde8c42f6858c2d/output.txt

Total script time: 19.53 mins

• Font tests: Passed
• Unit tests: Passed
• Regression tests: FAILED

Image differences available at: http://54.67.70.0:8877/fde8c42f6858c2d/reftest-analyzer.html#web=eq.log

[timvandermeij]

I agree that this approach is probably best. I'll make a follow-up ticket for the remaining intermittents and for re-enabling the FBF tests once Chrome stabilizes again.