This ensures that the devs who follows Debian instructions at gimp-web-devel
can build a full-featured GIMP and maybe we have a full-featured AppImage.
- Simplify three functions into 'bund_usr' (similar to 'bundle' in Win script)
- Add 'wipe_usr' function (similar to 'clean' in Win script)
- Verbose the steps of AppImage making
- Log (go)appimagetool bundling of deps (not only the the .appimage making)
- Change AppRun to be more clear about our .appimage compatibility
This refactoring is not perfect but is good enough to anyone understand
how our AppImage is made just reading the script.
This should be enough to not "annoy" developers with repetitive ninja and
not too relevant appimagetool output. This makes easier to devs to look at
the runner log without needing to revert 9653e50e (the revert/split would
make us lose .appimage in some custom Deb pipelines and 50% of ccache hits).
This adds a bit more verbosity to the installer script while omitting some.
The version params (e.g. gimp_version) are dropped since now config.h is
mandatory (this is a natural conclusion of the generation of assets).
'matrix' would save a lot of code and facilitate maintenance but,
according to my tests in a branch that I already deleted, this
doesn't work for our needs considering GitLab CI limitations.
This makes the bundling script an "extension" of the building script, similar
to what Cmake-based projects does on Windows. By the way, this is inline with
the recent changes which clarified that bundling is just a finalization of the
building process when a bundle is aimed. See: d09a2a6f, 2dc6f411 and 9d86492b
Despite the good intentions, 4f965557 makes Debian contributors who
follows gimp-web-devel instructions to download more deps than needed
and makes unclear what are the crossroad deps. Let's fix it.
This makes crossroad and msys2 scripts clone without the "_" prefix, which
will improve quality of life of most contributors that just clone them.
Just for consistency, also remove the "_" prefix from other pices of code.
The original script required a specific WinSDK version. Now, it autodetects.
This is useful if we need to move to another runner and welcome locally too.
Also, downloading the arm64 artifact was mandatory to get a x64 msix, which
is silly. Now, packagers are free to package to either arm64 or x64 alone.
This is useful locally but could be useful on CI too when some arch fail(?).
The ARTIFACTS_SUFFIX is being dropped on building scripts because:
1) This will make possible to further simplify Installer scripts in other commit
2) There is no script that uses multi-arch _build/_install at same time on CI
However, there are two use cases: to build Debian and flatpak; and, on Win,
to build on CLANGARM64 and CLANG64 (?). But probably they are pretty niche.
I suppose that people who build on Debian (or other dev-oriented distro)
willn't want to mantain deps in two sys prefixes (pkg and GNOME runtime)
+ two gimp prefixes (out of sandbox and sandbox) due to huge storage use.
Why someone would want to build with emulation on Win ARM64 I don't know.
Anyway, people that know how to do this stuff probably can change the .sh.
3) When building locally, the contributor doesn't need to know the arch at all.
Indeed, without this suffix, the scripts are inline with gimp-web-devel and
prevent some first-sight confusion when reading them that I've seen on IRC.
4) These arch suffixes can be understood as '_install-*' being distributable
which is not true. So, without this suffix we could make more clear
what is a package (when GitLab fix the glob bug in 'expose_as' someday).
---
Despite .gitlab-ci.yml not being a script, it needed to be touched too
because cross scripts depends on Debian jobs due to gimp-data MR.
The numbering is now inline with actual the order of jobs on CI
to make easier to understand what the numbers represent.
To understand why bundling is number 2 see: d09a2a6f and 2dc6f411
The storing of Windows scripts was changed to make easier to
call them, to better convey that they work outside CI and
to be consistent with other build/ subdirectories.
- Dependencies not present in GNOME runtime are now built in deps stage
This makes easier to follow the progress of the overall pipeline and
to know how much time was spent on each stage
(like crossbuilds, the artifact size is brutal but only lasts 2 hours)
- babl and GEGL build now have output in GitLab runner, unlike others deps
This makes clearer to spot if something goes wrong in these crucial deps
- GIMP is still built in its stage but now alone, like Debian and Windows
This makes possible to retrigger only this job when runner errors occours,
without needing to start monolithically the deps+gimp build from scratch
Also:
- babl, GEGL and GIMP now have meson-log.txt artifact like other builds
- dist job now have all commands self-contained on its script (needs to
be merged to be tested according to my tests)
Restores the build log artifact which was lost partly due to my fault.
The log is very useful in the rare cases of breaking the build system.
So, following our experience, let's save it for, at minimum:
- Native Linux
- Native Linux flatpak: will be adressed later in another commit
- Cross Win
- Native Win (aarch64)
If we were in the first days of our CI (so script bugs are common),
the scripts "debug mode" (bash -x) would be crucial, but we are not.
To be fair, running scripts in "debug mode" is pretty harmful today:
1) the produced logs consume our quota more and more (e.x.: adding a
bit more complicated .sh function can double the output size); and
2) the long white logs tends to be specially hard to read by the way.
Following 0199faac
We use arm64 runners for the installer job to avoid queues, but...
llvm-objcopy is well know to lack parity and one sympthom is the
'invalid SymbolTableIndex' warning when handling x86 debug symbols.
Completing d09a2a6f, since the dist job wasn't added to MR pipelines that time
Note: this doesn't change the actual behavior. The job will upload to nightly
repo only when triggered from schedule pipelines. I repeat, nothing changed.
What is new is that the .flatpak bundle now is an artifact from dist job, since:
- Despite the name, it isn't a real bundle, so no need to be only in gimp stage
- In dist, it reduces storage use because only the .flatpak file lasts 8 days,
not the .tar and other files from gimp job which lasts 2 days (5495483c)
Previous changes broken local building courtesy and probably DirectInput too.
Noticed that the universal variables can not work in some contexts, fixed.
Also, done some little cleanups.
This make them easier to understand and to run local, as we do with native ones
(They can't be run local yet, further changes will try to acomplish this a bit)
Reintroduce pkg caching for crossroad. The possibility of using it is one
in thousands since we don't have distributed cache, but it's better than
rely totally on MSYS2 mirrors, which can be be extremely sick sometimes.
file plug-in testing.
The name and location of the xml results will be the same as for the
log file, but with the extension replaced by .xml.
Just some basic support is added which adds failed testcases when
a plug-in has a failure result, which should be the most common
case of failures.
Not all cases where we make errors in our test setup will generate
failed testcases yet, although they should show up in the number of
failures.
Disabled tests won't show up as skipped tests.
We also don't track time.
The move to Clang in 85ed2847 didn't take into account caching,
Now, this is fixed or, better saying, reintroduced(?): e545116b
Also, flatpak-builder output was confined to the .log file to make
possible checking the end of the CI output, which was being cut.
Copied over from the original separate work in:
https://gitlab.gnome.org/Wormnest/gimp-file-plugin-tests
After that further improved and changed and added more file format
tests.
Added meson.build files to integrate it in our build, but we do not
install it for releases.
This regression was added by c064148a and it's similar to fb7a9954. Now, that
we are using 'interruptible: false' in custom rules, this is probably fixed.
Also, organize a bit better some pipelines with official GitLab .yml wizardy.
I was hesitating to use "<<: *" since this creates a new layer of complexity,
but this was the better way and the trick have a distinct marking by the way.
For our two "development" sub-stages, the expiration is as follows:
The 'dependencies' stage artifacts continues to expire in 2 hours.
Now, all 'gimp' ones expires in 2 days (to avoid time zone limbo).
For "stage"/tests and "production"/dist, the expire time depends
on the source of the CI pipeline. If the artifact is oriented to a
'short-span' source (e.g. MRs and commits), it expires in 2 days.
If is oriented to a 'LONG-span' one (e.g. web, schedule), 8 days
(we choose 8 to have a +1 day artifact just in case if something
goes wrong in the weekly schedule day, which isn't rare actually).
Partially reverts af79bbe0 (regarding .debug "extraction" in cross builds)
Now, we call the split debug script from the main bundling script, which makes
similar to our macOS .app bundling script. This cleans a bit of code in .yml
and make things clearer to the mere mortals in the scripts.
Native arm64 and x64 TWAIN binaries are not built anymore. No need to
discard at dist time what doesn't exist. (32-bit TWAIN not affected)
Also, start to using our own ARM64 runners to do the dist. This is more
reliable since the shared x64 runners can (and indeed) cause long queue.
They were generating a distracting output in CLANG* shells, as noted by
@lillolollo in a comment from MR: Infrastructure/gimp-web-devel!65
In the process, make AppImage and Windows (native) scripts use these
variables, without hardcoding the same variables from .yml anymore.
Creating a temporary config directory for the in-build GIMP (run as a tool or
for unit-testing) is not done as a build target anymore, but in the
in-build-gimp.sh script as a unique temp directory, then cleaned out on exit.
This has a few advantages:
- It is properly cleaned out once the build ends (instead of leaving a full
config dir as trash inside the build dir).
- It is not reused from one build to another (with risk of carrying bugs and
issues over).
- Every use of the in-build GIMP will have its own config directory, and in
particular when they are called in parallel.
As a side update, make sure that all `gimp_exe` runs depend on
`gimp_exe_depends`.
The new dist job can produce two artifacts: a *.msixupload file
ready for submission and/or a *.msixbundle only for testing.
See the commit about the script for more details.
Jehan