DevOps

Use GitLab to do .NET 4.X CI/CD

Dennys發表留言

This site is moved to GitHub, please visit the new URL: https://dennys.github.io/en/doc/devops/gitlab-dotnet4-ci-cd/

This site is moved to GitHub, please visit the new URL: https://dennys.github.io/en/doc/devops/gitlab-dotnet4-ci-cd/

This site is moved to GitHub, please visit the new URL: https://dennys.github.io/en/doc/devops/gitlab-dotnet4-ci-cd/

Precondition

  1. You already have a GitLab (GitLab SaaS or GitLab self managed)
  2. You want to build .NET 4.X, if you want to build .NET core or .NET 5 or above, you can use docker to run GitLab Runner.

Step 1 – Install Git client

Download and install Git for Windows. Portable version is ok, but because Git Runner is a Windows service, please add git.exe into PATH.

Step 2 – prepare .NET build environment

Download and install Visual Studio. (Please let me know if I can build a .NET build environment without Visual Studio, thanks.)

Step 3 – Install GitLab Runner

  1. Download GitLab Runner
  2. Rename gitlab-runner-windows-amd64.exe to gitlab-runner.exe
  3. Get the token to run GitLab Runner
  4. Generate GitLab Runner configuration file (config.toml)
    1. Run the command gitlab-runner.exe register --url https://gitlab.com/ --registration-token $REGISTRATION_TOKEN
  5. Modify config.toml
    1. Because Gitlab Runner doesn’t suppoer Windows shell after version 13, you need to use PowerShell. If you use Power Shell, please open config.toml and rename ‘pwsh’ to ‘powershell’. (If you use Power Shell Core, you don’t need to do anything)
    2. If you use a portable Git, you need to modify $env:Path of PowerShell.
    3. If you find your log is garbled (ex: your Windows server is not English version), please add “chcp 65001″ to change the encoding to UTF-8.
  6. Register GitLab Runner as a Windows service and start it.
    1. Run the command gitlab-runner install to install as a Windows service.
    2. Run the command gitlab-runner start to start the service.
  7. config.toml sample => remember, if you modify the file, please execute gitlab-runner.exe restart to restart GitLab runner.
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "GitLab Runner 007"
  url = "https://gitlab.com/"
  token = "******************"
  executor = "shell"
  shell = "powershell"
  pre_clone_script = """
      chcp 65001
      $env:Path += ";C:\\Gitlab\\PortableGit\\cmd"
  """
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

If you always see the Credential Helper Selector, please choose “no helper" and “Always use this from now on".

Step 4 – Write .gitlab-ci.yml

Edit .gitlab-ci.yml

.gitlab-ci.yml sample

stages:
    - build
    - test

build:
    stage: build
    script:
        - "dotnet build"
    artifacts:
      paths:
        - .\test

test:
    stage: test
    script: 
        - "dotnet test"

Step 5 – Start to build/test/deploy code (local machine)

Change directory to the location of .gitlab-ci.yml and execute this command

gitlab-runner.exe exec shell build

Step 6 – Start to test CI/CD (local machine)

If everything is ok, you can commit .gitlab-ci.yml and GitLab should run it automatically.

SAST (Static Application Security Testing)

GitLab can check your source code for known vulnerabilities, unfortunately, it only support Linux container, Windows containers are not yet supported. (reference: https://docs.gitlab.com/ee/user/application_security/sast/)

GitLab Checkmarx SonarQube integration

Dennys發表留言

Purpose: when we commit code to GitLab, we want

. (the next step is to integrate the scan result to SonarQube)

  1. GitLab sends the code to Checkmarx to scan.
  2. GitLab trigger SonarQube to sacn.
  3. SonarQube integrate Checkmarx ‘s report.

Official document: https://checkmarx.atlassian.net/wiki/spaces/SD/pages/169246832/SonarQube+Plugin+v8.5.0+and+up

Step 1. Download the plugin from here, it only supports SonarQube LTS version (for now, it’s 8.x)

Step 2. Configurea Quality Gate/Profiles of SonarQube for Checkmarx’s rules.

Step 3. Use GitLab to trigger Checkmarx scan and record the project name of Checkmarx.

Step 4. Configure Checkmarx data in SonarQube, you can reference here.

Step 5. Trigger GitLab CI again, you will see the following log in your SonarQube job

INFO: Sensor Import Checkmarx scan results to SonarQube [checkmarx]
INFO: Retrieving Checkmarx scan results for current module [Checkmarx plugin version: 2021.2.1]
INFO: Getting Checkmarx configuration data from sonar Database.
INFO: Resolving Cx setting: checkmarx.server.project_name
INFO: Forced authentication is enabled: Sonar credentials must be provided
INFO: Sonar server token is provided
INFO: Checkmarx credentials migration not needed
INFO: Sonar server token is provided
INFO: Resolving Cx setting: checkmarx.server.project_name
INFO: Forced authentication is enabled: Sonar credentials must be provided
INFO: Checkmarx server version [9.2.0.41015]. Hotfix [24].
INFO: Logging into the Checkmarx service.
INFO: Connecting to https://your.checkmarx.server/
INFO: Initializing Cx client [2020.2.4.NO.SCA]
INFO: Checkmarx server version [9.2.0.41015]. Hotfix [24].
INFO: Logging into the Checkmarx service.
INFO: full team path: \CxServer\\Team1
INFO: preset name: All
INFO: ---------------------------------Get Last CxSAST Results:--------------------------------
INFO: Waiting for server to generate xml report. 4990 seconds left to timeout
INFO: Checkmarx High vulnerabilities: 3
INFO: Checkmarx New-High vulnerabilities: 0
INFO: Checkmarx Medium vulnerabilities: 23
INFO: Checkmarx New-Medium vulnerabilities: 1
INFO: Checkmarx Low vulnerabilities: 142
INFO: Checkmarx New-Low vulnerabilities: 7
INFO: Checkmarx scan link: https://your.checkmarx.server//CxWebClient/ViewerMain.aspx?scanId=1000157&ProjectID=67

Step 6. You can see the Checkmarx issues in Sonar.

GitLab Checkmarx integration

Dennys發表留言

Purpose: when we commit code to GitLab, we want GitLab sends the code to Checkmarx to scan. (the next step is to integrate the scan result to SonarQube)

Official document: https://checkmarx.atlassian.net/wiki/spaces/SD/pages/1929937052/GitLab+Integration

Step 1. Edit .gitlab-ci.yml

include: 'https://raw.githubusercontent.com/checkmarx-ltd/cx-flow/develop/templates/gitlab/v3/Checkmarx.gitlab-ci.yml'

variables:
    CX_PROJECT: ProjectXXX #The project name you want to show in Checkmarx

Step 2. Edit GitLab CI/CD variables

Please remember to set GITLAB_URL and GITLAB_TOKEN, then Checkmarx will create GitLab issue for Checkmarx issues.

Step 3. Trigger build – Checkmarx analysis will be triggered when you create a merge request or you commit to master stream directly.

Suggest to use pure English for CX_TEAM, if you use non English (ex: Chinese), you can check the log and find it cannot find the team.

2022-01-12 02:51:22.748  INFO 11 --- [           main] c.c.s.s.CxService                         [x4DNMhOL] : Found team /CxServer/Team1 with ID 16

During the analysis, you can check the status from Checkmarx, it takes long time to analyze.


After the analysis, you can check the result in Checkmarx.

You can also check it from GitLab, it generates 1 report in merge request and it will create issues by category. The following is the reqport of merge request.

In the above report, there are 5 issues, it will creates 5 issues by file/by category.

In the issue, it shows the detail vulnerability information.