I really hate Jenkins for several non-objective reasons and it's not a classic hatred, more a life struggle where all episode begin by my signature on an employment contract. Most societies made the effort to open a private Gitlab but ask them to forget Jenkins and move into CI tools into SaaS is clearly unbelievable.
It is only a matter of time, and Opensource projects are already tied to this kind of platform. Most of this services are free for opensource projects and all volunteer contributors like me are very facilitated by these tools. All SaaS platforms are connected to GiHub via WebHooks, it's more or less 3 clicks for get ready to use them.
Before begin there's some global informations to know:
- This benchmark is Python oriented, more, it is Django oriented with a needed database backend and optionnaly a cache backend. I won't focus on other languages aspect and features but Selenium tests are in the scope.
- My primary test launcher is Tox, I use it for manage a matrix of tests with their dependencies and more, no need to make it with CI tool.
- Continious deployment is one of the primary feature of these tools, I will only treat about the continious integration side with test launching.
- Most of this SaaS applications seems to run on AWS infrastructure, if one of them don't I'll write it.
And in an idyllic world, I would like a tool able to:
- Launch my unit tests across different version of Python, Django and different DB backends
- Build project documentation
- Lint all source code
- Code coverage with threshold
- Integration with GitHub's pull requests
So let's test maximum of CI tools !
I think it is to Opensource what Jenkins is in private entreprise: Everyone should know and use it. Travis is the most used CI tool, its WebUI is relatively simple and its configuration is made by the famous
.travis.yml at your repository's root. This file represents a number of step defining a build:
Above this set of shell commands, a build is composed by jobs. Each job is an element of a matrix also defined in
.travis.yml and has the following dimensions:
- Language version
- + Manual inclusion
Matrix is a very useful tool as you could define a job for each version of Python with each version of Django with each database backend.
Next thing to list about Travis is their third services:
- Some of DBMS: MySQL, Postgres
- Some of NoSQL: MongoDB, Redis
- More and more: Docker, RabbitMQ
A bit like a sword strike into water, Travis supports build on OS X doesn't seem work with Python.
A well know CI tool with the same functionalities than Travis plus several options like debugging by SSH: You can connect to VM during and after build has succeed or failed.
CircleCI uses a
circle.yml file at base of your repository, it contains the following step for a build:
machine: Configure the test machine
- Checkout your code (not defined by user)
dependencies: Install dependencies
- Install database(s) (not defined by user)
test: Run test
All these steps are customizable by pre, overide and post hooks.
Parallelism is a bit more complicated than with Travis, there's no matrix system. Instead you allocate a number of containers availlable and must add a command for each of them. My tip for deal with this is explained here, I simply use as much tox's environment as I have containers.
With a tons of third services Circle has nothing to envy to Travis, a bit less but the most used are here.
There's no configuration file, all is helped by a great user experience. They has designed it for be easiest and more understandable as possible and a great tutorial helps to know quickly how to use it.
There is also no defined build step, user defines each step of his process with the following attributes:
- A name
- Trigger: automatic or manual
- Commands to execute
- Environment variables (secure or not)
- Secured files
- Number and size of worker needed
Snap-CI only deals with less than a dozen of SQL and NoSQL backends but it allows functionnal testing with browser or headless tools like Phantom-JS.
No configuration file too, all in WebUI. A build is really simple to create:
- Choose a language
- Choose a version, I think optionnal for me because, because all versions are pre-installed and are chosen by tox's virtual environment
- Create setup step with shell commands
- Create thread step with shell commands, several ones can be added for parallelization
- Create an optionnal post-thread step which be launched after each step
Semaphore CI's free plan allows only 2 parallel jobs which isn't enough for the kind of tests I want to make. My tests matrix is too big and I can only test 2 environments by build where Travis, for example, limits the number of active jobs but not the number of possible jobs.
About backends, most wanted databases are present: MySQL, PostgreSQL, Redis, MongoDB, Memcached or Cassandra. Semaphore CI still has some plus:
- Like CircleCI, you can connect to any jobs by SSH for debugging
- You can make pre-build file inclusion
- Schedule builds
- Selenium with Chrome and Firefox support
Last thing, I think it is great to signal Semaphore-CI isn't hosted by AWS but by Hetzner Online GmbH, a German hosting service and datacenter operator.
WebUI configuration again, no need to create YAML file. A build is simply defined by:
- Used language and its version
- One or more database or service to use
- Environmental variables
- Commands used for build
Unfortunately, this SaaS doesn't give the ability to create jobs matrix, you can define only one configuration per project. A second disadvantage is the number of third services, only 5, MySQL, PostgreSQL, MongoDB, Redis and Memcached.
The most simple and understandable CI tool, it is very comparable to Semaphore less some features. You must only configure setup and test commands in textareas. Both will be launched and test fails if a command fails. Nothing more simple.
This CI supports:
- SSH connection into test environment
- Few third services but the most wanted
- Fast job launching: It took 17 seconds to make the job against 1 minute for Travis
Parallelism is not free I just test the trial of 14 days, it is in continuity of their simplicity, you only add tabs at top of your test commands.
Where CodeShip become more complicated (but not too much) is at Docker integration, documentation is present, clear and simple but feature is only available for real customers with dollars.
AppVeyor has the singularity of make operations under Windows environment. For configure builds you have the choice to use an
appveyor.yml, their web-based configuration or both. A build is defined by:
And like a great CI tool AppVeyor allows simple jobs matrix simply with environmental variables groups.
All seems to be great in the best world but do not forget you're in a Windows environment:
- No shell command, let's play with Batch and PowerShell
- No easy_install, pip or etc
Luckily there are already templates available on GitHub. Copy/paste and it becomes easy to install all Python versions and setuptools under PowerShell.
As they offer 14 days of trial, I tested it, but Solano's pricing begins at $15/month. It requires a
solano.yml file for configuration, it has the following steps named here hooks:
boot: After repository checkout
worker_setup: Parallelized setup
post_setup: Global run after all setup
post_worker: Parallelized post test
post_build: Global run after all post_worker
For Python application, this service purpose 3 ways for test parallelization:
- nosetest or pytest: You define a list of files to use and their launching are parallelized
- command lines: You simply define a list of shell commands to parallelize
Real build matrix like make CircleCI and Travis is now only in beta and seems operationnal with Python. It allows to create jobs for each selected Python version.
When I wrote this article Solano didn't support (yet) tox, I made a request to their services and I think it will be implemented. For now I can't parallelize on Python or Django versions without hack.
This CI tool uses a
shippable.yml file which is 100% compatible with a
.travis.yml. It makes this service's behavior identical to Travis-CI but with notable technical differences.
- Where Travis took 10 minutes to make 10 jobs, Shippable took 1 hour for 4 jobs
- Test report is included and imported from a junit XML format
- Code coverage is included and imported from coverage XML format
You can also choose to run test from their container infrastructure by using a
DockerFile. Their offer a way to create advanced workflow for continious integration and deployment with Docker. You can pull and push images from Docker Hub, Google Container Registry, Quay.io or any other private repository.
Shippable also has a formation feature: You can create a full environment with several nodes for simulate real usecase or production environment. I think it is the edge of their Docker management, very useful but untested because not free.
It the one I like least because it doesn't have the needed complexity for my usage:
- There's no way to test build under serveral version of Python
- I can't use Tox because there's a bug where it can't list my env
- It is not able to manage other dependencies than
- Environment variables are managed under WebUI (not
wercker.yml) which not allow to build matrix based under them
Many details which make me dislike this service but it has its advantages. With the
wercker CLI tools and a Docker daemon you can launch localy your build and deploy step, this concept can speed up Docker developers workflow and ensure from dev to production environment is the same.
I saw on a webpage Magnum is
.travis.yml compatible, so I jump on their website, sign in and cp my configuration file to
.magnum.yml. Unfortunatlely all didn't work as expected for a lot of reason:
- This tool supports Python 2.7.3 but absolutely not documented
- It doesn't support build matrix, no Python version or environmental variable dimensions, only one configuration is launchable by build
- pip is not installed and aptitude timeouts
With a dozen of DB services and headless testing services it could be useful for project with simple and unique configuration. I think the best advantage of this CI tools is its free usage for all project, you only must have builds which not exceed 30 minutes.
A last thing to know, Magnum-CI is hosted by DigitalOcean.
Many SaaS CI purpose code coverage reports and Coveralls is entirely dedicated to this. It absolutely doesn't launch tests but gather coverage data about builds/jobs.
Coveralls usage is very simple if used on top of Travis. You must use
coverage as usualy and use
coveralls command in
after_script section. It will use your
.coverage and report to Coveralls API. This is the same for CircleCI and CodeShip.
If you don't use these service it's not hell, your must only add one environment variable COVERALLS_REPO_TOKEN findable in project's page.
In terms of features:
- Really great coverage report better than simple
- Aggregation of build's job allows you to know which environment doesn't cover lines
- Double threshold with among of coverage and decrease
- Great GitHub integration with status in pull requests
Another code coverage CI tool. For Python developers, Codecov seems to have nothing more than Coveralls but besides their file explorer they purpose a Web browser extension for display coverage directly on GitHub. See an example of this extension here.
Unfortunately, Codecov doesn't support (yet) coverage threshold and GitHub's pull request integration.
This SaaS tool measures the quality of project's code and returns a rate between 0.0 and 4.0. Its engine is based on:
- Code complexity: Essentially based on cyclomatic complexity (McCabe)
- Duplication: Identical or duplicate code
- Style: PEP8, PyFlakes and Co.
- Clarity: File and method's lenght, method's arguments
After analysis, all issues reported have weight and a letter grade from A to F is attributed to each files. Code Climate also has also a code coverage feature, like Coveralls it must be launched at end your CI build for send coverage data to Code Climate. I didn't use their code coverage because there's no Python tool for upload results.
Landscape is code inspector specialized in Python project, it is based on their open source tool Prospector, a combination of code quality tools:
- pylint: Base linter
- pep8: Code convention error
- pyflakes: Check for errors
- mccabe: McCabe complexity
- dodgy: Find secret in repository
- pep257: Docstring linter
- pyroma: Python package linter
- vulure: Find dead code in your library
All these commands are used for produce a rate in percent from 0 to 100. Issues are reported as:
- Errors: Probably buggy line code
- Smells: Design flaws
- Style: PEP8
Landscape also try to install all declared dependencies and create a small report about it. A
.landscape.yml could be write for configure severity of the tool, in fact it only does prospector command configutation.
This SaaS is also hosted by Hetzner Online GmbH. I really like this SaaS because they publish their base tools under GPL2 license all in Python.
This is a code quality inspector useful for me for track Python, JS and CSS files. It computes a letter grade from the following characteristics:
- Error Prone: Code shouldn't be in production
- Code Complexity: Complex code to be simplified
- Code Style: Language lint
- Unused Code: Dead code
- Security: Secrets stored in repository
- Compatibility: Compatibility across platform (browser)
- Documentation: Base documentation tests
- Performance: Issues that have performance impact
Codacy doesn't give a simple report, they have a great dashboard where all issues are explained well with a possible solution.
Gemnasium monitors your project's dependencies, all
requirements.txt files are read and one the following status is given:
- Green: Up-to-date
- Yellow: Not the latest
- Red: Outdated
Requires.io is also made for track project's dependencies but only designed for Python. It can find it in
setup.py. The resume is also better, all dependencies are sorted by files and the following attributes are given:
- Source code URL
- Website URL
- Asked version
- Is latest version
- Is up-to-date
- Python 3 or not
Sauce Labs is a PaaS for cross-browser testing. I don't test yet this platform and it will be the topic of an other entire article. It isn't bound directly to GitHub webhook but must be launched manually or by a CI tool.
- I launch ly unit tests with Travis-CI because it's the most used, supports most version of Python including PyPy and PyPy3 and matrix is easy to create and free.
- I could launch unit tests on Windows platform with AppVeyor
- I launch my functional tests with Sauce Labs
- I connect it to Coveralls and enhanced code coverage with pull requests integration.
- I inspect code quality with Landscape and web projects with Codacy
- And I monitor my dependencies with Requires.io
Links and references
- GitHub integrations page
- Travis-CI documentation
- CircleCI documentation
- Snap-Ci documentation
- Semaphore documentation
- Drone IO documentation
- CodeShip documentation
- AppVeyor documentation
- AppVeyor Python demo
- Solano CI documentation
- Shippable documentation
- Wercker documentation
- Magnum CI documentation
- Coveralls documentation
- Codecov documentation
- Code Climate documentation
- Landscape documentation
- Codacy documentation
- Gemnasium documentation
- Requires.io features
- Sauce Labs documentation