As you go about your daily business, moving JIRA tickets around and pushing code to Gitlab, most of you don't think about how it all works behind the scenes. It takes a bit of DevOps magic to make it all operate smoothly. As part of our internal development efforts, I was tasked with getting our source control, project management and continuous integration/delivery infrastructure off the ground. We chose, respectively, Gitlab, JIRA, and Jenkins. While this is a fairly standard setup and tutorials abound, we still ran into some niggling little issues, due to the simple fact that no two setups are truly the same.
This, then, is our contribution to the topic. It assumes three things:
1) You're running this on Linux: not a hard requirement, but all the file paths listed here pertain to Linux
2) You've installed everything necessary: we believe you can apt-get / yum / wget / curl just fine on your own
3) You've obtained an SSL certificate from Let's Encrypt or some other vendor: can't have encrypted, secure traffic without it
The scenario: We started with installing Gitlab Omnibus. You get Gitlab and are spared of installing and configuring a bunch of other stuff that comes in the package, including a web server to, well, serve Gitlab to users. But then we installed JIRA and realized – people can only reach it if they go to the IP: port for JIRA, as it comes with its own application server. We need a way to provide a uniform interface towards the user – we need a reverse proxy. We also want to access these services over a secure connection, but we don't want to configure each service individually. In other words, we need SSL termination.
This brings us to Nginx, which is typically used for both these roles. The idea is to present users with nice-looking URLs such as https://example.com/gitlab.
The front-facing Nginx then passes on (proxies) these requests to the actual servers running in the background, but terminates the SSL (well, TLS, but let's not pick nits) encryption before talking to them.
This means there are quite a few things to do: Add the proxied routes and the certificate to Nginx, disable SSL in Gitlab, JIRA and Jenkins, ask them to listen only to requests from the localhost as we don't want them to be reachable from the outside except via Nginx, and add any necessary configuration to make them work behind a reverse proxy.
Let's start with the Nginx side of things. We'll use "example.com" as the example domain (substitute as needed). Open your nginx.conf file (typically, it's in /etc/nginx), and first redirect HTTP to HTTPS in one server block:
server {
server_name example.com
listen 80;
return 301 https://$host$request_uri;
}
Now add the certificate information in another block:
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
server_name example.com
root /usr/share/nginx/html;
ssl_certificate "/etc/letsencrypt/live/example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/example.com/privkey.pem";
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES1$;
ssl_prefer_server_ciphers on;
}
The ssl_certificate and ssl_certificate_key are paths to your certificate and private key, which will be different for your setup. As a sidenote, if you've used certbot to register with Let's Encrypt and you used the --Nginx parameter (available via plugin), certbot will automatically modify your nginx.conf with the correct values (the options below ssl_certificate_key will be included through files rather than explicitly).
Now we can add the routes for Jenkins, JIRA and Gitlab (locations, as Nginx calls them) to the second server block. The crucial part is the proxy_pass option, which routes to the ports on localhost. You can set these to any port value; 8081, 8084, and 10987 are just some examples. Naturally, these must coincide with the ports in the configuration files for each service. Since all the routes are fairly similar, first we'll get the two nearly identical ones out of the way:
server {
location /jenkins {
proxy_pass http://127.0.0.1:8081;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 90;
proxy_http_version 1.1;
proxy_request_buffering off;
}
location /jira {
proxy_pass http://127.0.0.1:8084;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 500M;
}
}
Gitlab is a little different:
server {
location /gitlab {
proxy_pass http://127.0.0.1:10987;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Protocol https;
proxy_set_header X-Url-Scheme https;
proxy_set_header X-Forwarded-Ssl on;
proxy_read_timeout 90;
client_max_body_size 0;
gzip off;
proxy_http_version 1.1;
}
}
The most important difference is the third block, the SSL options. Gitlab Omnibus comes with its own Nginx server, and the SSL options must be properly handled on both sides.
Don't forget to run Nginx -t to test the configuration file for errors before starting the server.
Let's proceed to configuring Jenkins. Open the configuration file, located either in /etc/default/jenkins (Debian) or /etc/sysconfig/jenkins (CentOS), and set the following options:
JENKINS_PORT="8081"
JENKINS_LISTEN_ADDRESS="127.0.0.1"
JENKINS_ARGS="--prefix=/jenkins"
This sets Jenkins up to listen on port 8081, take only requests coming from the localhost, and listen on the prefix /jenkins.
Now let's take a look at JIRA
If you've installed the binary from Atlassian, your JIRA installation is in /opt/atlassian/jira, and you need the /conf/server.xml file. Comment out the other connectors, and add this one:
connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false"
keystorePass="changeit" keystoreType="JKS" maxHttpHeaderSize="8192"
maxSpareThreads="75"
maxThreads="150" minSpareThreads="25" port="8084"
protocol="org.apache.coyote.http11.Http11NioProtocol" proxyName="example.com"
proxyPort="443" redirectPort="8443" relaxedPathChars="[]|"
relaxedQueryChars="[]|{}^\`"<>"
scheme="https" secure="true" sslProtocol="TLS" useBodyEncodingForURI="true"/>
The salient attributes are port, proxyName and relaxedQueryChars. The first two are self-explanatory, and you'll input your own values. The last one you can copy as-is - JIRA tends to complain if you don't include it.
We're finally onto Gitlab
The gitlab.rb configuration file should be located in /etc/gitlab/:
external_url "https://example.com/gitlab"
gitlab_rails['gitlab_shell_ssh_port'] = 22
nginx['listen_addresses'] = ['127.0.0.1']
nginx['listen_port'] = 10987
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
"X-Forwarded-Ssl" => "on",
}
The gitlab_shell_ssh_port needs to be changed to mirror the one defined in sshd.conf. It's not a good practice to leave the default value, which you'd quickly learn upon checking the logs and seeing thousands of login attempts.
The rest of the options concern Gitlab's internal Nginx instance. The first two are again self-explanatory, and the last two are crucial for making it work with SSL.
There will always be little things and changes that you need to make to tailor these tools to your own purposes, but these instructions should set you well on your way towards a basic setup. Hurdles are an inevitable part of life. SSL can be finicky. Some setups appear strange at first – two instances of Nginx can lead to all sorts of confusing situations. XML configuration files will drive you to wonder why (oh why) people still use XML for configuration. The best thing you can do is to jot these down for future reference – and who knows, maybe a blog post down the line.
