[tex-live] Automated TeX Live Integration Testing
pander at users.sourceforge.net
Thu Sep 20 10:23:54 CEST 2012
On 2012-09-07 14:26, Heiko Oberdiek wrote:
> On Fri, Sep 07, 2012 at 12:50:33PM +0200, Pander wrote:
>> Would be nice that the people who told me to start working on this that
>> some feedback would be given.
>> On 2012-08-15 12:25, Pander wrote:
>>> Please review:
>>> tlit - TeX Live Integration Testing
>>> The goal of integration testing of TeX Live is to automatically confirm
>>> that an update did not break any major functionality. Detailed module
>>> testing of individual packagese is up to the package maintaineres. This
>>> is done in much more detail and is outside of the scope of integration
>>> testing. Integration testing offered here is merely a simply check if
>>> certain often used combinations of package can play together. The goal
>>> is to prevent that simultaneous updateting of multiple packages, which
>>> all passed their individual module tests without being aware that other
>>> packages are also being updated on the same time, break TeX Live.
Heiko, thanks for having a look at it.
> * It assumes that packages have individual module tests that are run on
> updates. Both is not true with few excaptions only.
Modules are tested by developers before uploading to CTAN, that I assume
but it is indeed unclear with which version of TeXLive this is done or
done at all. Actually I don't care, I just want to know if post upgrade
all is working. I will remove the assumption on the documentation.
> * To some degree package testing already involves other packages.
> There is no sharp separation line for integration testing.
True. I just want to prevent what prevent situation that result in
non-functional TeX Live installation as happen (but was fixed quickly)
> * I cannot find, how this is addressed:
> * There are quite a few TeX compilers with quite a few formats.
> * There are lots of classes and tons of packages.
> * Classes and packages provide none up to an endless number of options
> with multiple values and working modes.
> * The number of package combinations and loading order is overwhelming.
The tests currently in there are some major packages and engines I use.
If they do not work, that is a big enough signal for me. I would expand
the test cases only with incidents that happen in the future.
> * What is tested:
> * Errors: expected or unexpected
> * Warnings: expected or unexpected
> * Messages: expected or unexpected
> * Output: DVI/PS/PDF
> * log file, aux files, ...
Simply and only the result status at this time.
> * Which system to manage the tests?
Hope some has one available.
> * How to deal with the result? But report to package authors?
To be discussed and implemented.
> * ...
> Good testing is a huge task requiring lots of "(wo)man power".
My idea was to start something or give an impulse to this and have it
evolve by contributions from all. I have worked with many software
development environments that have a nightly build and subsequent
testing whenever changes have been made to the repository that day.
PS If some of you want a .tar.gz, please go to
> Yours sincerely
> Heiko Oberdiek
More information about the tex-live