All Articles

Using docker client in a CI environment via Fastlane

Using docker client in a CI environment via Fastlane

I’m currently working in Brubank’s iOS app (https://apps.apple.com/us/app/brubank/id1295202448). When testing, we use a server that we bootstrap locally. We have a mixed test base of unit and integration tests.

As our tests depend on the backend, some months ago I implemented (via scripting) the code to setup the server, that run in Bitrise (CI).

This was achieved via fastlane, and the part of getting the server up was somehow like this:

  desc 'Bootstrap the hub'
  lane :bootstrap_hub do |options|
    hub_path = options[:hub_path]

    set_go_env_variables
    install_casks
    install_brews
    start_services
    install_pips
    clone_hub_repo(hub_path: options[:hub_path], hub_branch: options[:hub_branch])

    Dir.chdir(hub_path) do
      sh("make")
    end
  end

  #example of those lanes
  private_lane :start_services do
    sh('brew services start rabbitmq')
    sh('brew services start postgresql')
    sh('brew services start elasticsearch')
    sh('brew services start redis')
  end

  private_lane :install_pips do
    sh("pip install idna chardet urllib3 faker requests")
  end

Each of those functions were needed to set the whole environment (i.e: install go, install python, install pips, start brew services, etc…).

While this worked, it clearly wasn’t the better solution. When the backend needed to change something about the setup, like the version of go, or new dependencies were added, we would need to change the script. B

However, the bigger issue came with when the backend decided to migrate to microservices. For us it was problematic, because there was a different repo with that microservice and the whole logic of bootstrapping that microservice would need to be scripted, which would take a huge amount of time.

While trying to figure out a solution, we needed to:

  1. Be able to boostrap the server on every build.
  2. Be able to choose a different branch from the server (i.e: If we are working in feature1 in the app, we need like to run our tests with feature1 server’s branch).
  3. Do this as performant as possible. Don’t forget that we need to build a iOS app while we do this whole setup

How did we solve these problems?

The first step was to talk with our awesome infra team (one of the best I’ve worked with) on how we could achieve this. They helped me by explaining how our server could be built with Docker and they implemented this using docker & azure. One week later, I had the images available in docker to try!

But, what is docker (https://www.docker.com/)?

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.

So my task was to be able to run Docker locally (both while developing in my machine and in Bitrise).

I think someone could benefit of my process of thought, so the steps I planned were:

  1. Be able to use docker locally in my machine. It involved getting a security key, and it would give me a glimpse of how much I would need to script.
  2. Be able to run tests locally using the docker’s hub.
  3. Be able to have docker installed in a machine in bitrise.
  4. Be able to pull an image while running a bitrise job.
  5. Be able to run tests with using the docker’s hub in bitrise.
  6. Be able to pull specific branches of our server, not just our master.

Solving the mentioned points

(1) and (2) were achieved with fairly easiness, I literally sat near the infra team for the whole process. However, with (3) “Be able to have docker installed in a machine in bitrise” was a different story.

If you want to pull a docker image, you would explicitly need docker’s app installed (the one that looks like a whale) The problem in CI is that it is fairly complicated to launch docker headless (i.e: without a desktop, from command line). Docker doesn’t run natively in OSx: it runs natively on linux. The trick that docker app uses is a tool named hypervisor. From the docs:

Docker for Mac is a native MacOS X application that embeds a hypervisor (based on xhyve), a Linux distribution and filesystem and network sharing that is much more Mac native. You just drag-and-drop the Mac application to /Applications, run it, and the Docker CLI just works.

But we cannot simply drag and drop while running in CI.

From https://github.com/docker/for-mac/issues/3567:

Docker.app is not running as a service because it runs in user space.
It is a gui app that launch underlying components so you need to be logged in.

There is no way to launch them without the gui in the current version.
Sorry for that.

So, no way of launching docker without a GUI with the current version of docker.

However, there was some light. According to https://github.com/docker/for-mac/issues/2359, one could do this:

brew cask install docker
sudo /Applications/Docker.app/Contents/MacOS/Docker --quit-after-install --unattended
nohup /Applications/Docker.app/Contents/MacOS/Docker --unattended &

and it would work. So I tried that method.

Building on top of the previous solution

I continued scripting using the same idea as above. It’s important to note that the docker version needs to be one that supports launching it via unattended without crashing. I tried with the current version but it didn’t work. I then found that there was an older version that did work.

So the scripts look like the following:

  desc 'Bootstrap the hub'
  lane :bootstrap_hub do |options|
    sh('brew install docker docker-compose')
    # This is the last version of docker that supports installing headless (from terminal).
    # https://github.com/janeczku/dotfiles/blob/master/install-software.sh
    sh('curl -O -sSL https://download.docker.com/mac/stable/31259/Docker.dmg')
    sh('open -W Docker.dmg && cp -r /Volumes/Docker/Docker.app /Applications')
    sh('sudo /Applications/Docker.app/Contents/MacOS/Docker --quit-after-install --unattended')

    ...
  end

with this docker was installed.

There’s also a delay for when docker is turning on. So we need to to wait until docker is up. We can query docker state with docker stats --no-stream. When it returns a successful response, then docker is up:

  WAIT_FOR_DOCKER_DELAY = 5 * 60

  desc 'Bootstrap the hub'
  lane :bootstrap_hub do |options|
    sh('brew install docker docker-compose')
    # This is the last version of docker that supports installing headless (from terminal).
    # https://github.com/janeczku/dotfiles/blob/master/install-software.sh
    sh('curl -O -sSL https://download.docker.com/mac/stable/31259/Docker.dmg')
    sh('open -W Docker.dmg && cp -r /Volumes/Docker/Docker.app /Applications')
    sh('sudo /Applications/Docker.app/Contents/MacOS/Docker --quit-after-install --unattended')

    docker_pid = fork do
        Signal.trap("INT") { exit }
        sh('nohup /Applications/Docker.app/Contents/MacOS/Docker --unattended &')
    end

    response = ''
    start_time = Time.now
    loop do
      UI.message("Waiting until Docker is up...")
      begin
        response = sh("docker stats --no-stream")
      rescue => ex
        response = ''
      end
      if Time.now - start_time > WAIT_FOR_DOCKER_DELAY
        UI.user_error! "Timeout while waiting for docker app to be running!"
      end
      break if response != ''
      sleep(3)
    end

    UI.success("Docker is up!")
  end

There we have docker already working in a headless environment. We need to login to docker to pull the images:

## Replace with your credentials
sh("echo #{ENV["DOCKER_PASSWORD"]} | docker login <docker-registry> -u <docker-user> --password-stdin")

Now we are authenticated into docker.

We can easily script starting and stopping the server:

  desc 'Starts the hub server'
  lane :start_hub do |options|
    sh('docker-compose pull')
    sh('docker-compose up -d')
  end

  desc 'Stops running hub'
  lane :stop_hub do |options|
    sh('CONFIGS="--dev /hub/hub/config_bitrise.toml" docker-compose down')
  end

With this and some more scripting, we could achieve almost every point except (6).

(6) Be able to pull specific branches of our server, not just our master.

To achieve that point, after talking with the infra team, they made two endpoints available:

  1. Force the id of the docker image to match the name of the branch in the server.
  2. Be able to query if a branch is already built. If so, just pull it.
  3. Be able to trigger a new image build from the ci.

Considerations

  1. Note that there is a docker-compose.yml that depends on your configuration. I’m not adding here, but you will need it.

  2. Beware with threading in fastlane. I wrote this to kill docker thread after everything finish running (specially errors):

  desc 'Kill process'
  lane :kill_process do |options|
    pid = options[:pid]
    # Note: 2 corresponds to SIGINT, which is == ctrl + c
    sh("lsof -ti:8081 | xargs kill -2")
    Process.kill("INT", pid)
    Process.wait
  end
  1. Sometimes, quite randomly, docker fails in CI with an error:
[18:29:18]: $ docker stats --no-stream
[18:29:19]: ▸ Analytics: Sent backendStartingVm event.
[18:29:48]: ▸ DockerState: stopped
[18:29:49]: ▸ Terminator: Supervisor has failed, shutting down: Supervisor caught an error: one of the children died: com.docker.driver.amd64-linux (pid: 2467)
[18:29:54]: ▸ error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.39/version: EOF

This is WIP, but I’m trying to catch that error and in that case attempt to initialize docker again. Something like this:

    docker_pid = initialize_docker
    response = ''
    start_time = Time.now
    loop do
      UI.message("Waiting until Docker is up...")
      begin
        response = sh("docker stats --no-stream")
      rescue => ex
        # Sometimes it randomly fails, so we initialize docker again.
        if ex.to_s.include? "DockerState: stopped"
          UI.message "Restarting Docker app ..."
          kill_process(pid: docker_pid)
          docker_pid = initialize_docker
          UI.success "Docker app restarted!"
        end
      end
      if Time.now - start_time > WAIT_FOR_DOCKER_DELAY
        UI.user_error! "Timeout while waiting for docker app to be running!"
      end
      break if response != ''
      sleep(3)
    end

    desc 'initialize docker and return pid'
    private_lane :initialize_docker do |options|
      docker_pid = fork do
        Signal.trap("INT") { exit }
        sh('nohup /Applications/Docker.app/Contents/MacOS/Docker --unattended &')
      end
      docker_pid
    end

Published Mar 26, 2020

Software Engineer, Mentor and Philosopher