Self-hosted GitLab Installation and Setup

Montreal

Installation

  1. Install self-hosted Gitlab on the server acting as manager node for the docker swarm (server-1).

  2. Setup self-signed certificates to be copied into the secrets directory (see below) on both servers.

  3. Disable user creation to avoid undesired users follow these instructions.

  4. Talk to whoever manages your network to have them add your new hostname (e.g., cpip.server-2.imagerie.user-vms.cqgc.hsj.rtss.qc.ca) to the DNS.

  5. Follow these instructions 1 or instructions 2 and reconfigure gitlab to accept self-signed certificates. #. Note: use this information to create the keys where here we are using an IP, but ideally we would be using a DNS associated to that IP.

  6. Enable a container registry with the same self-signed certificate.

Docker Swarm GitLab Installation

  1. Follow the steps for the the creation of selfsigned certificates that was previously outlined. Or not, if you already have certificates for the required hostnames.

  2. For Montreal’s setup we will deploy GitLab, StoreSCP in the manager node (server-1), and the gitlab-runners in a worker node (data/processing server-2).

  3. Create a docker swarm manager node to deploy GitLab, StoreSCP, and connect MinIO to the UniC object store.

    sudo docker swarm init --data-path-port 9789 --advertise-addr 127.0.0.1

    Warning

    Be aware of the issues with docker swarm in a VMWare virutal machine.

  4. SSH into the worker node (processing server-2) and run the following command with the information obtained from the previous command.

    sudo docker swarm join --token TOKEN server-1.internal.imagerie.user-vms.cqgc.hsj.rtss.qc.ca:2377

    You can then verify the status of the swarm on the manager node running:

    sudo docker ps

    Warning

    You must use the hostname for server-1 on VLAN4 here identified by the “internal” keyword.

  5. Create an attachable docker overlay network. The subnet/gateway was defined to avoid potential issues with the port ranges used by the University of Calgary.

    docker network create --driver=overlay --attachable cpip-network --subnet=192.11.0.0/16 --gateway=192.11.0.2

    You can list the docker networks and get more specific status information on the cpip-network using:

    sudo docker network ls
sudo docker network inspect cpip-network

  6. Clone the UNF repository ni-dataops/stack.

    Note

    If you are using self-signed certificates, you might use the selfsigned-cert branch, for this branch custom docker images will need to be created before stack deployment. Update: use montreal branch.

  7. Create the necessary docker secrets.

    Under deploy there is a script called create_directory.sh and generate_secrets.sh. Run the directory creation script, and then generate the secrets. Make sure that you have a folder under ~/ni-dataops/stack/secrets where your secrets are located in raw text form i.e. gitlab_root_password.

    Make sure that the generate_secrets.sh script generates all the required secrets for each service, as can be seen in the docker compose config yaml file. Copy these onto both servers:

    secrets:
  gitlab_root_password:
    file: ./secrets/gitlab_root
  gitlab_token_local:
    file: ./secrets/gitlab_local
  gitlab_token_remote:
    file: ./secrets/gitlab_remote
  cert:
    file: ./secrets/bundle.crt
  key:
    file: ./secrets/cert.key
  minio_pass:
    file: ./secrets/minio_pass
  minio_conf:
    file: ./secrets/mc.conf
  dicom_bot_token:
    file: ./secrets/dicom_token
  s3_id:
    file: ./secrets/s3_id
  s3_key:
    file: ./secrets/s3_key
  ssh_passphrase:
    file: ./secrets/passphrase

    Note that here the gitlab_token_local is the personal access token you must generate from your self-hosted gitlab, whereas gitlab_token_remote is the one you generate from the UNF gitlab instance. The bundle.crt and cert.key are created when generating the self-signed certificate info using:

    openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout cert.key -out bundle.crt

  8. Make sure that the docker-compose file point the service deployment to the manager node using the constraints and attached to the right network.

    deploy:
placement:
    constraints:
    - node.hostname == manager-node.ca

       networks:
      - cpip_network
networks:
cpip_network:
   external: true

  9. Do the modifications necesary to set your hostname and run the command:

    sudo GITLAB_HOME=/srv/gitlab/ docker stack deploy -c docker_compose.gitlab.yml cpip

    Important

    In docker swarm, in order to mount a volume to a container, such volume must exist. This is not necessary using docker compose where directories are created if missing.

    Note

    You can find information on how to change password using the terminal in this disscusion.

    #You will need to do this through the ruby console
user = User.where(id: 1).first
user.password = 'your secret'
user.password_confirmation = 'your secret'
user.state = 'active'
user.save!
exit

  10. More documentation on how to automatically set the instance wide CI/CD gitlab variables to come.

  11. There will be a way to standardize/automatically set the instance wide CI/CD gitlab variables using python scripts and json configuration files from the ni-dataops/stack/gitlab_server/config UNF repository.

    #This way of creating ci-variables will involve running something like this

python3 create_gitlab_variable.py ci-variable.json

  12. Follow the previous steps to configure gitlab.

Calgary

Docker Swarm GitLab Installation

  1. Follow the steps for the the creation of selfsigned certificates that was previously outlined. Or not, if you already have certificates for the required hostnames.

  2. For Calgary’s setup we will deploy GitLab, StoreSCP, and MinIO in the manager node, and the gitlab-runners in a worker node (data server).

  3. Create a docker swarm manager node to deploy GitLab, StoreSCP, and MinIO.

    docker swarm init --dport 9789

    Warning

    Be aware of the issues with docker swarm in a VMWare virutal machine.

  4. SSH into the worker node (processing server) and run the following command with the information obtained from the previous command.

    docker swarm join --token TOKEN --advertise-addr <IP-ADDRESS-OF-WORKER-1> <IP-ADDRESS-OF-MANAGER>:2377

  5. Create an attachable docker overlay network. The subnet/gateway was defined to avoid potential issues with the port ranges used by the University of Calgary.

    docker network create --driver=overlay --attachable cpip-network --subnet=192.11.0.0/16 --gateway=192.11.0.2

  6. Clone the UNF repository ni-dataops/stack.

    Note

    If you are using self-signed certificates, you might use the selfsigned-cert branch, for this branch custom docker images will need to be created before stack deployment.

  7. Create the necesary docker secrets.

    sudo docker secrets create name-of-secret secret-file
# secret-file can be any text file containing the needed information.
# OR
echo "xxxxxxxxxx" | docker secret create name-of-secret -
# make sure to remove the entry from the server's history

  8. Make sure that the docker-compose file point the service deployment to the manager node using the constraints and attached to the right network.

    deploy:
placement:
    constraints:
    - node.hostname == manager-node.ca

       networks:
      - cpip_network
networks:
cpip_network:
   external: true

  9. Do the modifications necesary to set your hostname and run the command:

    sudo GITLAB_HOME=/srv/gitlab/ docker stack deploy -c docker_compose.gitlab.yml cpip

    Important

    In docker swarm, in order to mount a volume to a container, such volume must exist. This is not necessary using docker compose where directories are created if missing.

    Note

    You can find information on how to change password using the terminal in this disscusion.

    #You will need to do this through the ruby console
user = User.where(id: 1).first
user.password = 'your secret'
user.password_confirmation = 'your secret'
user.state = 'active'
user.save!
exit

  10. More documentation on how to automatically set the instance wide CI/CD gitlab variables to come.

  11. There will be a way to standardize/automatically set the instance wide CI/CD gitlab variables using python scripts and json configuration files from the ni-dataops/stack/gitlab_server/config UNF repository.

    #This way of creating ci-variables will involve running something like this

python3 create_gitlab_variable.py ci-variable.json

  12. Follow the previous steps to configure gitlab.

Debbugging

  1. Allow a new ssh port in the system can be achieved. Follow this post for more information.

  2. There is an error when using docker swarm for the deployment this post mentions how to solve it.

    # All you need to do is add the following configurtion to the gitlab runners config in /etc/gitlab-runner/config.toml
[[runners]]
#....
[runners.docker]
   pull_policy = ["if-not-present", "always"]
   #...

Direct GitLab Installation

We follow this installation guide for installing gitlab in centos/redhat 8, it also works for redhat 9. It is imporant to make the following considerations when following the steps.

  1. Disable user creation to avoid undesired users follow these instructions.

  2. Secure GitLab Server with self-signed certificates.

    1. Create a self-signed certificate. Click here for the creation of a self signed SSL certificate on centos or redhat.

      Note

      This link also contains information on how to eliminate the problem: x509: certificate relies on legacy Common Name field, use SANs instead. This will help you create a new certificate that will contain a subject alteranative name.

      For instance for an self-signed certificate cpip.ahs.ucalgary.ca you would do the following.

      openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/gitlab/ssl/cpip.ahs.ucalgary.ca.key -addext "subjectAltName = DNS:cpip.ahs.ucalgary.ca" -out /etc/ gitlab/ssl/cpip.ahs.ucalgary.ca.crt

    2. Talk to whoever manages your network to have them add your new hostname (e.g., cpip.ahs.ucalgary.ca) to the DNS. Wheter or not this is important will depend on who will need to have access to gitlab’s interface.

    3. Follow these instructions 1 or instructions 2 and reconfigure gitlab to accept self-signed certificates.

  3. Enable a container registry with the same self-signed certificate.

    1. You can use the same certificate which means that your registry will be deployed on the same domain name but with a different port. For instance: cpip.ahs.ucalgary.ca:5050.

    2. To do this you need to follow these instructions to configure the container registry under an existing gitlab domain.

    3. Follow the steps on this link to install self-signed certificates and make sure they are trusted by adding the public part of the PEM .crt certificate to the /etc/gitlab/trusted-certs/

      1. This is how you can get the PEM public certificate.

        sudo openssl x509 -inform PEM -in /etc/gitlab/ssl/cpip.ahs.ucalgary.ca.crt -pubkey -noout > /etc/gitlab/trusted-certs/cpip.ahs.ucalgary.ca.crt

      2. You need to make sure that you also add the certificate to docker daemon and to the system-wide trusted certifictes folders like so:

        1. First to docker: follow the steps suggested in this post.

          1. Copy the cerficate you created in Create a self-signed certificate into the /etc/docker/cert.d folder, create if it does not exist.

            sudo cp /etc/gitlab/ssl/cpip.ahs.ucalgary.ca.crt /etc/docker/cert.d/cpip.ahs.ucalgary.ca:5050/ca.crt

          2. Add the hostnames to the insecure registries json file in /etc/docker/daemon.json. I added both with and without port but I am almost positive you only need the cpip.ahs.ucalgary.ca:5050

            {
"insecure-registries" : [ "cpip.ahs.ucalgary.ca","cpip.ahs.ucalgary.ca:5050" ]
}

        2. You also need to make sure that your system trusts the created certificate by following these instructions. These are specific o RedHat 8 follow a simillar guide for your OS.

          sudo cp /etc/gitlab/ssl/cpip.ahs.ucalgary.ca.crt /etc/pki/ca-trust/source/anchors/cpip.ahs.ucalgary.ca.crt
sudo update-ca-trust extract

      Note

      You can find information on how to change password using the terminal in this disscusion.

      #You will need to do this through the ruby console
user = User.where(id: 1).first
user.password = 'your secret'
user.password_confirmation = 'your secret'
user.state = 'active'
user.save!
exit

Configuration

After installation, there are additional configurations required before the pipeline is ready to process images.

  1. First, install gitlab-runner following the tutorials, and create the minimal number of instance-wide (can be accessed by jobs triggered from any repository, even if created after the creation of the runners) runners required.

  2. Create an empty new project called ni-dataops.

  3. Clone the ni-dataops ci-pipelines and containers repositories from https://gitlab.unf-montreal.ca/ni-dataops and push upstream to you self-hosted gitlab. Access (token-access) to this repository should be allowed from other repositories, this will permit newly created repositories containing data to access the processing pipelines.

    Note

    This can be done in the CI/CD settings of the gitlab project in the interface.

    git clone https://gitlab.com/cal_cpip/ni-dataops.git
cd ni-dataops
git remote add <name-of-remote> <url-of-self-hosted-gitlab-project, for instance https://cpip.ahs.ucalgary.ca/ni-dataops.git>
git push -u <name-of-remote> main

    Note

    Check branch permissions to make sure you can push up to it.

  4. Create some users which will be necessary to run some of the task like DICOM to BIDS conversion, processing, etc.

    1. bids_bot = Admin level so it can access all repos

    2. dicom_bot = Admin level because its token need to have elevated privileges to use with the GitLab API.

  5. Install MinIO in you data server following this guide.

  6. Some instance-wide variables need to be setup in order for CI/CD pipelines to use then even when new repositores are added after.

    Note

    To do this you need login into the self-hosted GitLab’s admin area. There, you will need to navigate to the settings > CI/CD > Variables.

    1. BIDS_API_TOKEN = access token for the bids_bot

    2. BOT_SSH_KEY = this key is generated from the gitlab-runner from the bids runner

      Note

      This is the private key starting with ——-something——- and ending with ———–end————. It should be generated from inside the runner instance.

      Note

      Additionally, the public part of the key added need to be added to bids_bot profile ssh_keys.

    3. GIT_BOT_USERNAME = bids_bot

    4. GIT_BOT_EMAIL = bids_bot@ahs.ucalgary.ca

    5. S3_SECRET = S3 password set in the minio installation

    6. SSH_KNOWN_HOSTS = created copying the output of ssh-keyscan <IP of your self-hosted gitlab> into the value of the variable.

      Note

      This variable needs to contain Host and IP of the self-hosted Gitlab

Debbugging

  1. Allow a new ssh port in the system can be achieved. Follow this post for more information.

  2. There is an error when using docker swarm for the deployment this post mentions how to solve it.

    # All you need to do is add the following configurtion to the gitlab runners config in /etc/gitlab-runner/config.toml
[[runners]]
#....
[runners.docker]
   pull_policy = ["if-not-present", "always"]
   #...