“How Jenkins Builds the Netflix Global Streaming Service ?”​

What is Jenkins?

Jenkins is an open source Continuous Integration server capable of orchestrating a chain of actions that help to achieve the Continuous Integration process (and not only) in an automated fashion.

Jenkins is free and is entirely written in Java. Jenkins is a widely used application around the world that has around 300k installations and growing day by day.

It is a server-based application and requires a web server like Apache Tomcat. The reason Jenkins became so popular is that of its monitoring of repeated tasks which arise during the development of a project. For example, if your team is developing a project, Jenkins will continuously test your project builds and show you the errors in early stages of your development.

Why Jenkins?

Jenkins is a software that allows continuous integration. Jenkins will be installed on a server where the central build will take place. The following flowchart demonstrates a very simple workflow of how Jenkins works.

Along with Jenkins, sometimes, one might also see the association of Hudson. Hudson is a very popular open-source Java-based continuous integration tool developed by Sun Microsystems which was later acquired by Oracle. After the acquisition of Sun by Oracle, a fork was created from the Hudson source code, which brought about the introduction of Jenkins.

Benefits of Jenkins as a CI tool

Prevent Assignment of Work — Various kinds of work will occur at the time of development, but if you leave nothing in the build, you are unknowingly becoming an adolescent. Such work becomes a work that everyone can do by automating it as a job of Jenkins.

One Can Manage Old Versions — In software development, function addition and modification are repeated many times, and many versions are created. Sometimes it may be necessary to verify with older versions. In such a case, if you build with Jenkins, you can easily download the old version that is being managed.

Information Is Easy to Obtain — Because the number of users is large and the community is active, it is easy to deal with information when it is easy to obtain information and problems occur. Also, many books have been published, and getting know-how is relatively easy. With emerging CI tools this will not work.

Major Features

High Versatility — One can run scripts of various kinds/languages in any environment such as Linux / Windows / Mac.

Extensive Trigger — It can be executed with various triggers such as “commit of version control system”, “scheduling by cron-like mechanism”, “when another build is completed”, “request with specific build URL”, etc.

Extensibility by Plug-In — It can extend its functionality by creating/using plugins. It also features a lot of plugins.

Continuous Integration(CI)

Continuous Integration is a development practice that requires developers to integrate code into a shared repository at regular intervals. This concept was meant to remove the problem of finding later occurrence of issues in the build lifecycle. Continuous integration requires the developers to have frequent builds. The common practice is that whenever a code commit occurs, a build should be triggered.

Continuous Delivery (CD)

Continuous delivery is the ability to make changes of all types-such as new features, configuration changes, error fixes, experiments-into production in a safe and efficient manner using short work cycles.

Continuous Deployment (CD)

Continuous deployment, also known as continuous implementation, is an advanced stage of continuous delivery that the automation process does not end at the delivery stage. In this methodology, every change that is validated at the automatic testing stage is later implemented at the production stage.


Netflix, Inc. is an American over-the-top content platform and production company headquartered in Los Gatos, California. Netflix was founded in 1997 by Reed Hastings and Marc Randolph in Scotts Valley, California.

Jenkins at Netflix

Not Just a Build Server

  • Monitoring of our Cassandra clusters
  • Automated integration tests, including bake and deploy
  • Housekeeping and monitoring of :
  1. the build / deploy infrastructure
  2. our AWS environment

Jenkins Statistics

  • 1600 job definitions, 50% SCM triggered
  • 2000 builds per day
  • Common Build Framework changes trigger 800 builds
  • 2TB of build data
  • Over 100 plugins

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Scaling Challenges — and How We Met Them

Jenkins Scaling Challenges

  • Thundering herds clogging Jenkins, downstream systems
  • Making global changes across jobs
  • Making sure plugins can handle 1600 jobs
  • Testing new Jenkins and plugin versions
  • Backups

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Building a New Master

  • Jenkins build watches http://mirrors.jenkins-ci.org/war/ latest/jenkins war and downloads the new Jenkins WAR file
  • Download latest versions of our installed plugins (from a list stored externally)
  • Create an RPM containing the Jenkins webapp
  • Bake an AMI with the RPM installed under Netflix Tomcat

Job Configuration Management

Configuration Slicing plugin

  • Great for the subset of config options that it handles
  • Don’t want to extend it for each new option

Job DSL plugin

  • Set up new jobs with minimal definition, using templates and a Groovy based DSL
  • Change the template once to update all the jobs that use it

More Maintenance

  • System Groovy scripts to clean up old job configs and associated artifacts:
  • Disable jobs that haven’t built successfully for 60 days
  • Set # of builds to keep if unset
  • Mark builds permanent if installed on active instances
  • Delete unreferenced JARs from Artifactory

Keeping Slaves Busy

  • System Groovy scripts to monitor slave status and report on problems
  • The Dynaslave plugin: slave self-registration

Further Reading

http://tech blog netflix.com http://www.slideshare.net/netflix https://github.com/netflix http://jobs.netflix.com

Published By

Sheikh Aafaq Rashid

Originally published at https://www.linkedin.com.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sheikh Aafaq Rashid

Sheikh Aafaq Rashid

Hello, my name is Sheikh Aafaq Rashid . I'm currently learning Devops related tools. Arth Learner | trainee @ Linux World informatic pvt ltd