A microservice for facilitating the creation, storage, and retrieval of IIIF manifests and collections.
There are just a few prerequisites that must be installed, and configured correctly, in order to build Fester:
- Java Development Kit (JDK): version 11 or greater
- Docker: version 19.03 or greater
- Maven: version 3.6 or greater
- AWS CLI [Not required, but useful]
These packages may also be available through your system's package repository. If they are, it's better to install from that source so that they will be kept up to date for you.
You will also need an account on AWS and have the ability to create IAM accounts and S3 buckets.
Fester uses an S3 bucket for back-end storage. To be able to run the project's tests, several configuration values must be supplied:
- fester.s3.bucket
- fester.s3.access_key
- fester.s3.secret_key
- fester.s3.region
These values can be set as properties in your system's Maven settings.xml file (or be supplied on the command line at build time).
The project builds an executable Jar that can be run to start the microservice. To build the project, run:
mvn package
This will put the executable Jar in the target/build-artifact directory.
To generate the site's documentation, run:
mvn site
This will generate the documentation in the target/site directory.
The project contains unit, functional, and integration tests, with controls on how to control which tests are run. In order to run the functional and integration tests, the build machine must have a working Docker environment. Setting up Docker on your machine will depend on the type of machine you have (e.g., Linux, Mac, or Windows). Docker's documentation should be consulted on how to do this.
When running the build using the 'package' phase (as described above), only the unit tests are run. If you want to run all the possible tests, the project can be built with:
mvn integration-test
or
mvn verify
This will run the functional, feature flag, and integration tests, in addition to the unit tests. If you want to skip a particular type of test but still run the 'verify' phase, you can use one of the following arguments to your Maven command:
-DskipUTs
-DskipITs
-DskipFTs
-DskipFfTs
The first will skip the unit tests; the second will skip the integration tests; the third will skip the functional tests; and, the fourth will skip the feature flag tests. They can also be combined so that two types of tests are skipped. For instance, only the functional tests will be run if the following is typed:
mvn verify -DskipUTs -DskipITs
When running the integration and functional tests, it may be desirable to turn on logging for the containers that run the tests. This can be useful in debugging test failures that happen within the container. To do this, supply one (or any) of the following arguments to your build:
-DseeLogsFT
-DseeLogsIT
-DseeLogsFfT
This will tunnel the container's logs (including the application within the container's logs) to Maven's logging mechanism so that you will be able to see what's happening in the container as the tests are being run against it.
You might also want to adjust the logging level on the tests themselves. By default, the test loggers are configured to write DEBUG logs to a log file in the target directory and ERROR logs to standard out. To change the log level of the standard out logging, run Maven with the logLevel argument; for instance:
mvn -DlogLevel=DEBUG test
If you want more fine-grained control over the logging, you can copy the src/test/resources/logback-test.xml file to the project's root directory and modify it. A logback-test.xml file in the project's home directory will be used instead of the standard one in src/rest/resources if it's available. That hypothetical file has also been added to the project's .gitignore so you don't need to worry about checking it into Git.
It is sometimes useful to run a single test (instead of the whole test suite). The Surefire Maven plugin allows for this, but it's worth noting that when a single test is run in this way the test suite's pre-configured system properties are not picked up from the plugin's configuration. To work around this, a dev who wants to run a single test must supply the necessary properties theirself. For example, if one wanted to run the functional test that checks that missing images get a placeholder image in the manifest, the command to do that would be:
mvn integration-test -Dtest=MissingImageFT -Dfester.s3.bucket=iiif-fester -Dfester.placeholder.url="https://iiif.library.ucla.edu/iiif/2/blank" -Dfester.logs.output=true
You would want to supply your own values for fester.s3.bucket and fester.placeholder of course. This command will spin up the Docker container that the functional test is run against, but it will only run the MissingImageFT test, skipping all the integration and other functional tests in the suite.
You can run a development instance of Fester by typing the following within the project root:
mvn -Plive test
Once run, the service can be verified/accessed at http://localhost:8888/fester/status. The API documentation can be accessed at http://localhost:8888/fester/docs
There are two ways to debug Fester:
- Debugging the tests. This enables the developer to step through both the test and application code as the test suite runs.
- Debugging a running instance. This enables the developer to step through the application code as they interact with the HTTP API.
The following setup instructions were tested with Eclipse IDE 4.14.0 (2019-12).
From within Eclipse:
- Create a new debug configuration
- In the top-level menu, select Run > Debug Configurations...
- In the pop-up window:
- Create a new configuration of type Remote Java Application
- Set Name to something like
Fester (JDWP server for containerized instances created by test suite) - In the Connect tab:
- Set Project to the Fester project directory
- Set Connection Type to
Standard (Socket Listen) - Set Port to
5556 - Set Connection limit to
16 - Check Allow termination of remote VM (optional)
- Create another debug configuration *
- In the top-level menu, select Run > Debug Configurations...
- In the pop-up window:
- Create a new configuration of type Maven Build
- Set Name to something like
Fester (debug test suite) - In the Main tab:
- Set Base directory to the Fester project directory
- Set Goals to
integration-test - Set Profiles to
debug - Set User settings to the path to a
settings.xmlthat contains your AWS S3 credentials
- Run the debug configuration created in Step 1 **
- Run the debug configuration created in Step 2 **
* As an alternative to step 2 (and 4), run the following from the command line (after completing steps 1 and 3):
mvn -Pdebug integration-test
** If you're doing this for the first time, you may need to bring back the pop-up window where you created the configuration in order to invoke it. Otherwise, you can use toolbar buttons, or hotkeys Ctrl F11 (Run) or F11 (Debug).
This procedure will start an instance of Fester with port 5555 open for incoming JDWP connections.
From within Eclipse:
- Create a new run configuration ***
- In the top-level menu, select Run > Run Configurations...
- In the pop-up window:
- Create a new configuration of type Maven Build
- Set Name to something like
Fester (debugging mode) - In the Main tab:
- Set Base directory to the Fester project directory
- Set Goals to
test - Set Profiles to
runDebug - Set User settings to the path to a
settings.xmlthat contains your AWS S3 credentials
- Create a new debug configuration
- In the top-level menu, select Run > Debug Configurations...
- In the pop-up window:
- Create a new configuration of type Remote Java Application
- Set Name to something like
Fester (JDWP client) - In the Connect tab:
- Set Project to the Fester project directory
- Set Connection Type to
Standard (Socket Attach) - Set Host to
localhost - Set Port to
5555 - Check Allow termination of remote VM (optional)
- Run the new run configuration created in Step 1
- Run the new debug configuration created in Step 2
*** As an alternative to step 1 (and 3), run the following from the command line:
mvn -PrunDebug test
and then proceed with steps 2 and 4.
A Locust test file is included, it only tests PUTs of manifests. If you wish to run the test, you need to have Locust installed, and then run the following command from the src/test/scripts/locust folder:
locust --host=url-of-the-server-you-are-testing
For example, if you wish to run a Locust test against a dev instance on your own machine, you would enter:
locust --host=http://localhost:8888
To prevent accidentally pushing commits that would cause the CI build to fail, you can configure your Git client to use a pre-push hook:
ln -s ../../src/test/scripts/git-hooks/pre-push .git/hooks
We pin the versions of packages that we install into our base image. What this means is that periodically a pinned version will become obsolete and the build will break. We have a nightly build that should catch this issues for us, but in the case that you find the breakage before us, there is a handy way to tell which pinned version has broken the build. To see the current versions inside the base image, run:
mvn validate -Dversions
This will output a list of current versions, which can be compared to the pinned versions defined in the project's POM file (i.e., pom.xml).
Festerize may be used to interact with Fester, as an alternative to the built-in CSV upload form.
We use an internal ticketing system, but we've left the GitHub issues open in case you'd like to file a ticket or make a suggestion. You can also contact Kevin S. Clarke at [email protected] if you have a question about the project.