How I reduced CircleCI time from 50 mins to 8

January 29, 2024 (10mo ago)

TLDR; write jobs as commands and have everything run as a single job

The scenario

Assume the following scenario: You have a very well written CircleCI conf file where you have a number of jobs (4 or more. I had about 10). Your jobs are well structured and follow Single Responsibility & DRY principles. Some of the jobs run in parallel, while some of them are sequential. Your CircleCI pipeline takes a lot of time and you want to reduce that.

In my case the pipeline needed about 50mins to complete

The problem

In such a scenario you have to wait for each job to download the docker image, checkout the code, install packages, run build scripts, and more. Maybe you have jobs persisting to workspace and then other jobs having to attach to that workspace. On top of this maybe you have a monorepo and the workspace is very large (a few GBs). Attaching 3GBs needs 3mins on each own. On top of that you need to account for CircleCI job switch times, start times and more. All of these things add up and end up taking a lot of time.

Solution

Now, the good news is that you can greatly optimise this. The naive approach to do this is to include everything (or as much as you can) in a single job so you avoid time consuming things like downloading docker images multiple times, installing same packages multiple times, attaching to workspace as all steps have access, waiting for circleci to switch jobs, and more. The problem with this approach is that is very difficult to maintain the conf file as it is a huge list and there's no concept of breaking things down.

commands to the rescue. CircleCI offers a commands section. This can be called within a job like steps. But contrary to steps, commands are reusable and you can still maintain a good structure and a well written file, by maintaining Signle Responsibility and DRY principles.

Here's an example command:

version: 2.1
 
commands:
  say_hello:
    description: "A simple command to echo Hello World"
    steps:
      - run: echo "Hello World"
 
jobs:
  build:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - say_hello # Use the custom command
 
workflows:
  version: 2
  build_and_test:
    jobs:
      - build

Example

Here's an example of a file that can greatly benefit from the technique explained in this article.

Let's assume we have a monorepo with 1 backend service and 2 frontend services (frontend_service_1, frontend_service_2). The following CircleCI config, builds the monorepo, then persists files to worskpace, and then deploys backend, frontend_service_1, and frontend_service_2.

version: 2.1
 
jobs:
  build_monorepo:
    docker:
      - image: cimg/node:latest
    steps:
      - checkout
      - run: 
          name: Install Packages
          command: npm i
      - run:
          name: Build Monorepo
          command: npm run build
      - persist_to_workspace:
          root: .
          paths:
            - node_modules
            - frontend_service_1
            - frontend_service_2
 
  deploy_backend:
    docker:
      - image: cimg/node:latest
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Deploy Backend
          command: |
            # Add deployment commands here
            echo "Deploying backend..."
 
  deploy_frontend_service_1:
    docker:
      - image: cimg/node:latest
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Deploy Frontend Service 1
          command: |
            # Add deployment commands for service 1 here
            echo "Deploying frontend service 1..."
 
  deploy_frontend_service_2:
    docker:
      - image: cimg/node:latest
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: Deploy Frontend Service 2
          command: |
            # Add deployment commands for service 2 here
            echo "Deploying frontend service 2..."
 
# Define workflows
workflows:
  version: 2
  build_and_deploy:
    jobs:
      - build_monorepo
      - deploy_backend:
          requires:
            - build_monorepo
      - deploy_frontend_service_1:
          requires:
            - build_monorepo
      - deploy_frontend_service_2:
          requires:
            - build_monorepo

In the above example the time required to download each docker image, and to attach the workspace takes a lot of time. Depending on the size of each project might take from a few seconds to a few minutes. Now, we can rewrite the above as follows using commands and make it much faster. (we no longer benefit from parallelism while deploying, but in our pipeline the amount it takes for the workspace to be attached is 3 mins, and this methods ends up being much faster)

version: 2.1
 
commands:
  job-build_monorepo:
    description: "Build the entire monorepo"
    steps:
      - run: 
          name: Install Packages
          command: npm i
      - run:
          name: Build Monorepo
          command: npm run build
 
  job-deploy_backend:
    description: "Deploy the backend"
    steps:
      - run:
          name: Deploy Backend
          command: echo "Deploying backend..."
 
  job-deploy_frontend_service_1:
    description: "Deploy frontend service 1"
    steps:
      - run:
          name: Deploy Frontend Service 1
          command: echo "Deploying frontend service 1..."
 
  job-deploy_frontend_service_2:
    description: "Deploy frontend service 2"
    steps:
      - run:
          name: Deploy Frontend Service 2
          command: echo "Deploying frontend service 2..."
 
# Define a single job using the commands
jobs:
  build_and_deploy:
    docker:
      - image: cimg/node:14.17.0
    steps:
      - checkout
      - job-build_monorepo
      - job-deploy_backend
      - job-deploy_frontend_service_1
      - job-deploy_frontend_service_2
 
# Define workflows
workflows:
  version: 2
  all_steps:
    jobs:
      - build_and_deploy
LinkedIn