Reducing CD Pipeline complexity in large organisations


On your journey to developing a scalable continuous delivery pipeline, complexity increases rapidly. How do companies like Amazon, Facebook and Tesla cope with this?


We offer a 3-step plan that will guide you through the process of reducing complexity and enabling new teams to use Continuous Delivery 3.0 from the start. For this we use Jenkins, an exemplary customizable Continous Delivery platform, but the concepts should transfer to all other platforms as well.

  1. Create A Collective Library

    It is common that Jenkins plugins are simply insufficient. In that case, you’ll have to write a script yourself. These scripts add up over time and to avoid duplicate work, organisations should store these scripts into a collective library.

    In addition, you should strive to convert all Jenkins servers, pipelines and jobs into configuration files and add these to the collective library.

  2. Building Pipelines Without scripting

    Upstream pipelines can trigger a sequence of downstream jobs. By developing minor downstream job-scripts and storing them in the collective library, other developers can concatenate these jobs in their upstream Continuous Delivery pipeline. This approach allows for pipeline creation with limited scripting knowledge and code-digging.

    Unfortunately there is one limitation, the pipeline creator requires understanding of each script’s parameters. This may result in delayed pipeline development and could slow down productivity of both the pipeline and job-script creators.

  3. Automated script parameters

    To solve this problem and reduce the entry threshold even more, pipeline creators can add a minor piece of code that automatically loads these parameters from a file. This extra file requires job developers to carefully think about script user-friendliness resulting in a better collective scripts library overall.

    At this point your organisation has created a community of developers that actively writes user-friendly jobs for themselves and others. Through quality work and culture, developers should be motivated to look at the powerful collective library first, before they write a new script themselves.