Catalog management
This page documents how to setup and use your own catalogs. Please contact us if you have questions!
All instructions are command line calls based on the Unix syntax - Windows users have to use backslashes for local paths.
Catalog types
We distinguish between direct and request catalogs. Both are based on GIT and can be uploaded to for example GitHub or GitLab. They differ in how new or updated solutions are handled via the command album deploy [solution] [catalog]
:
Changes to a direct catalog will be immediately accessible to anyone who uses this catalog. The changes are directly pushed to the
main
branch of the catalog.Changes to a request catalog have to be reviewed. The changes are pushed to a branch. We provide a GitLab CI script automatically creating a merge request to the catalog repository which also supports Zenodo DOI generation.
We suggest using the different types in these scenarios:
For rapid development use a direct catalog - for example for local, personal, or temporary catalogs.
For federated catalogs and versioned solutions with DOIs use a request catalog.
How to create and use a local catalog
Use this command to create a new catalog:
album clone template:catalog [catalog-dir] [catalog-name]
This will create a bare git repository including the basics of a direct catalog in [catalog-dir]
.
Add it to your collection by calling this:
album add-catalog [catalog-dir]
Write your new solutions anywhere on disk and deploy them into the catalog with this command:
album deploy [solution-path] [catalog-name]
Upgrade the catalog in your local collection:
album update [catalog-name]
album upgrade [catalog-name]
How to track solution development [within the catalog repository]
We recommend making a separate repository for developing your solutions, next to the catalog repository (see our default and default-dev repositories as an example - default
is a catalog which was created via the clone
command demonstrated below, default-dev
does not have a fixed structure, it’s just the source from where we deploy solutions into the catalog)
One can also add the source files of solutions to the catalog repository - here is an example:
Add your solution source files for example to [catalog-dir]/src/
:
[catalog-dir]/src/groupA/solution1/solution.py`
[catalog-dir]/src/groupA/solution2/solution.py`
[catalog-dir]/src/groupB/solution1/solution.py`
Add them to the repository using the default git commands. This is how you would deploy these solutions into the catalog:
album deploy [catalog-dir]/src/groupA/solution1 [catalog-name]
How to create a catalog with a website representation
We provide a Gatsby theme and a GitLab CI setup for automatically deploying a web representation of a catalog. This won’t work on GitHub, but the CI script can surely easily be adjusted to also work on GitHub.
Proceed as described in How to automatically share solutions to GitLab / GitHub, but instead of template:catalog
clone template:catalog-gatsby
(or instead of template:catalog-request
use template:catalog-request-gatsby
). The webste will be deployed using GitLab pages
It’s also possible to add the website feature to an already existing catalog. The whole Gatsby website procedure is not documented well yet and will be improved.
Rapid prototyping
While developing a solution, developers and testers might not always want to deploy new versions of a solution - we recommend attaching -SNAPSHOT
to the version
one is working on (i.e. 0.1.0-SNAPSHOT
prior to releasing 0.1.0
).
In order to test changes to the run method of a solution rapidly without having to uninstall and reinstall a solution, use the --override
flag after deploying the solution:
album update [catalog-name]
album upgrade [catalog-name] --override
The --override
flag will check for each changed solution if it is already installed, and update the local cache of the solution content in this case.
Now album run [solution]
will use the updated solution file. Be aware that any changes to the setup
method of the solution will not be processed,
including changes to the environment of the solution. In order to apply these, the solution needs to be uninstalled and reinstalled.
DOI support for solutions
We provide tools to automatically upload solutions to Zenodo and add the respective DOI to the solution during solution deployment. This requires more steps when creating a catalog and these steps still need to be documented properly.