windows-terminal/build/Helix
Mike Griese 8f4c63e81b
Plumb test failures through to github (#15831)
This does two bits:
1. correctly marks our tests as failed in xUnit, so that AzDo will pick
up that the tests have failed.
2. Actually intentionally mark skipped tests as skipped in xUnit. We
were doing this accidentally before.
3. Add a CI step to log test failures in a way that they can show up on
GitHub


Probably regressed around #6992 and #4490.

### details

#### Part the first
We were relying on the MUX build scripts to convert our WTT test logs to
xUnit format, which AzDo then ingests. That script we used relied on
some WinUI-specific logic around retrying tests. They have some logic to
auto-retry failed tests. They then mark a test as "skipped" if it passed
less than some threshold of times. Since we were never setting that
variable, we would mark a test as "skipped" if it had _0_ passes. So,
all failures showed up on AzDo as "skipped".

Why didn't we notice this? Well, the `Run-Tests.ps1` script will still
return `1` if _any_ tests failed. So the test job would fail if there
was a failure, AzDo just wouldn't know which test it was.

#### part the second
Updates `ConvertWttLogToXUnitLog` in `HelixTestHelpers.cs` to understand
that a test can be skipped, in addition to pass/fail. Removes all the
logic for dealing with retries, cause we didn't need that.

#### part the third
TAEF doesn't emit error messages in a way that AzDo can immediately pick
up on which tests failed. This means that Github gives us this useless
error message:

![image](https://github.com/microsoft/terminal/assets/18356694/3be6de00-22e1-421c-93d4-176bd2be4cab)
That's the only "error" that AzDo knows about. 

This PR changes that by adding a build step to manually parse the xUnit
results, and log the names of any tests that failed. By logging them
with a prefix of `##vso[task.logissue type=error]`, then AzDo will
surface that text as an error message. GitHub can then grab that text
and surface it too.

### Addenda: Why aren't we using the VsTest module
as noted in
https://github.com/microsoft/terminal/pull/4490#issuecomment-583104982,
the vstest module is literally 6x slower than just running TAEF
directly.
2023-08-15 09:50:15 -05:00
..
ConvertWttLogToXUnit.ps1 Plumb test failures through to github (#15831) 2023-08-15 09:50:15 -05:00
HelixTestHelpers.cs Plumb test failures through to github (#15831) 2023-08-15 09:50:15 -05:00
OutputTestErrorsForAzureDevops.ps1 Plumb test failures through to github (#15831) 2023-08-15 09:50:15 -05:00