Getting started with Concourse for VMware Tanzu on PhotonOS

Last week VMware announced a huge set of product releases, focusing on VMware Tanzu, vSphere 7.0 with Kubernetes inside, and VMware Cloud Foundation 4.0

A large part of the Tanzu product offering involves products and services that came with the acquisition of Pivotal.

One of the Tools that VMware has inherited from the Pivotal ecosystem, is a powerful CI/CD pipelining tool called Concourse.

Continuous Integration and Continuous Delivery (CI/CD) is a set of techniques and way of thinking, that enables developing and delivering code fast, which a high amount repeat-ability through automation.

Over the years, various tool sets that enable CI/CD have become popular, such as Jenkins, but there are many more. In fact VMware already has their own CI/CD tool, VMware Code Stream
.CI/CD tooling can now often be found as part of Git platforms such as Github and Gitlab.

Concourse CI has been the de-facto tool to use in Pivotal Cloud Foundry environments, and while Concourse itself is an open-source project, Pivotal is and, as VMware, remains much involved in its community.

Concourse is an open-source project, and its home is here
https://concourse-ci.org/

As part of the VMware Tanzu offering; take a look at the official VMware page for this here: https://tanzu.vmware.com/concourse
It is from this page you can also get at all the Pivotal resources regarding Concourse

Concourse can be used to automate anything. It doesn’t execute logic itself, but is used to pipeline other bits of executing code, such as scripts. It executions those scripts inside containers that it spins up as needed. This execution mechanism makes it quite different from other CI/CD tools. It also makes its results highly reproducible. The concourse server itself only executes your pipelines, it doesn’t save any state itself.

If you want to learn more about how Concourse works, check out the links at the bottom of this post.

In this post I will cover the setup of a PhotonOS VM on VMware Workstation 15.5, and then the basic setup of Concourse running in docker with an example pipeline.

Setting up our Photon 3.0 VM

I am using VMware Workstation 15.5 to set up this VM. We are using the Minimal ISO image
https://github.com/vmware/photon/wiki/Downloading-Photon-OS

To Execute, Concourse needs RAM, for demo purposes, 2Gb should be enough. You may find you need more. Out-of-memory errors can surface in strange ways in Concourse.. such as Docker runtime errors complaining about broken pipes.

When using the PhotonOS ‘minimal’ ISO, ssh login is not permitted to anyone by default. So we must first turn this on if we wish to use ssh.



Restart sshd to enable the ability to log in


I have unfortunately had a lot of problems with the systemd-resolved stub resolver. As you can see below, using it seems to fail in my lab causing “temporary failure in name resolution”, and I have not been able to figure out yet why this happens.

Changing my /etc/resolv.conf symlink to point to the static DNS file solves this issue.


I set IPtables to let ICMP in and out of the VM

Then we activate enable the docker daemon and turn it on

I then set up my login for dockerhub

Now lets install Git, because we want to be able to pull repositories onto our VM


We now need to install docker-compose. Grab the latest version from the official repo

curl -L https://github.com/docker/compose/releases/download/1.25.4/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Our VM is now ready for use. You might want to reboot it to check if everything keeps working.

Running Concourse in Docker

We are going to use the official Concourse project docker images using Docker-Compose.
https://github.com/concourse/concourse-docker

If you have Kubernetes running, or you use BOSH, it is also available as a helm chart or BOSH deployment:
https://docs.pivotal.io/p-concourse/v5/installation/

Concourse as a Docker deployment, consists of three containers: Postgres, the Concourse web frontend, and a worker node

Lets start by cloning the repo so we have all the files we need to work with


Now we need to run a small script in concourse-docker/keys that will generate the https keys that are needed for the 3 services to talk to each other

We must now change one thing in the docker-compose.yml
Because we are running concourse in a VM with a dedicated domain name, we must update the CONCOURSE_EXTERNAL_URL value to the name our VM can actually be resolved on.

Back in the main directory of the repo, we can now start Concourse by simply typing:

docker-compose up -d

( -d denotes that we want to run it as a daemon )

Docker will now pull the three images it needs and start the containers. It takes about 1GB of space for this.

Check the containers are running

We can now check to see if the UI is running on our VM, on port 8080

We can login using the default account ‘test’ with password ‘test’

There is not much to see here yet, as we have not yet loaded any pipelines into Concourse. We do that with the ‘fly’ command line tool.

The ‘fly’ cli tool

In order to interact with Concourse, we need to use the ‘fly’ tool. This is a single binary that you can download for mac, windows or linux, and enables us to manage our Concourse deployment and upload pipelines to it, among other things.

You can download the fly tool to our VM and run it there, or you can use it locally. In this case, I will run it from the Windows WSL Ubuntu environment.

You can download fly straight from web-interface. In fact this is quite important; the fly executable is tied to the specific version of Concourse you are using. It won’t work with an older version, its not backwards compatible.

If you find yourself having to deal with various versions of Concourse, you might want to keep various versions of the fly cli binary handly. One way to do this is to simply rename them to the various versions, like fly_580.

For working with WSL, I have a tools directory that I have added to my windows system path, that makes it available inside WSL

Fly can be used against different concourse servers. It saves these relationships as ‘targets’, that you are free to name anything.

To add a new target, and to login to current ones, we use the login command. The way concourse allows you to login is a little convoluted, it wants to do a token-exchange by having you visit a specific URL.


Copy-paste this URL into your browser.

If you are running your browser, and the fly command-line on the same machine, then Concourse will automatically send fly the token and automatically log you in.

If this mechanism does not work, you have to manually copy-paste the token from the webpage, into your terminal. In most terminals this will work ok, but I have had some trouble with putty sometimes. It may take you several attempts.

‘Teams’ are a way of dividing a Concourse server into different tenants. If you don’t specify a team, fly will log you in into the ‘main’ team.

Configuring a simple pipeline

We will now upload a simple demo pipeline to Concourse using fly. You can find it here:
https://github.com/thefluffysysop/Concourse , but it is merely a copy-paste from Stark & Wayne’s excellent Concourse Tutorial

You can clone it with this command: git clone https://github.com/thefluffysysop/Concourse.git

When setting a pipeline, fly will ask you to confirm the upload. If you are updating an existing pipeline with some changes, it will only show you the changes.

In the UI, our pipeline will appear. Visually, it contains just a single solid box. This is because our pipeline only contains one job.


We can click on our pipeline to see more details. Notice that it is paused.
We can press ‘play’ at the top right, this will ‘activate’ our pipeline, enabling any jobs we have to run.
The jobs will not run automatically however, they require specific triggers to start, or to be started manually.

We only have one job. We can start it manually by clicking into it, and pressing the ‘+’ button at the top right. This will start a new job execution, or in Concourse terminology, a ‘build’ Concourse will list the jobs that where triggered in the top bar and will number them. It will give them a color according to if they where executed successfully.

Jobs that are being run, will show in the pipeline screen as having yellow radiating lines around them, so you can easily see at which job your pipeline is.

A job consists of tasks, and our own job only had one task ‘hello world’. You can click on tasks as they are being executed to see their command line output.

When the job is complete, it turns green. In the output of the job, you can see the result of the command we specified in our task (inside our job, inside our pipeline). In this case its a just an echo, but it might as well have been a shell script or a remote ssh command. It could be anything.

Now what?

Concourse is very powerful, and can be used to automate basically anything.

In the future, I will be sharing my own experiments with Concourse as I learn to use it.

If you want to learn your way around Concourse and make your own first pipeline, I very much recommend Start & Wayne’s Concourse tutorial to get your started.
https://concoursetutorial.com/

The official Concourse wiki at Github, have an addition list of learning and tutorial resources
https://github.com/concourse/concourse/wiki/Tutorials

The next place is of course the pojects own documentation.
https://concourse-ci.org/docs.html – they also have a forum, a Discord server for live chatting.

Pivotal (now VMware) maintains their own documentation at
https://docs.pivotal.io/p-concourse/v5/ , but bare in mind this is more oriented to viewing concourse as part of the larger Tanzu/Pivotal ecosystem such the Pivotal Cloud Foundry and Pivotal Container Services products.

There are some great videos on youtube that explain how Concourse works, and show demo’s
https://www.youtube.com/results?search_query=concourse

I also made a Concourse channel in the VMware Code Slack, come talk to me there!

I hope this post helps you on your way playing with Concourse for the first time. There is a good chance you will see Concourse pop up in the VMware ecosystem more in the future. Give it a try, and see how powerful it can be.