@megabytelabs/codeclimate-flake8

v1.0.2

Published

2 years ago

A slim Flake8 and a CodeClimate engine container for GitLab CI

Downloads

0High
0Medium
0Low

ci codeclimate container docker dockerfile gitlab gitlab-ci mblabs megabytelabs pipeline professormanhattan virtualization washingtondc

A slim Flake8 and a CodeClimate engine container for GitLab CI

Overview

This project is the home of a Flake8 CodeClimate engine. It was built to integrate Flake8 results into the GitLab merge request web UI. This project also generates a standalone, slim instance of Flake8 that you can use to quickly run Flake8 in a CI/CD pipeline.

Flake8 is a wrapper around these tools:

Flake8 is also extendable with plugins. You can find a wealth of Flake8 plugins on the Awesome Flake8 page on GitHub.

Standalone

If you are interested in using the tool and have no need for CodeClimate integration, you can get the latest, slim, and versioned images without CodeClimate-related code by removing the codeclimate- string from the image name. For example, if the image is named megabytelabs/codeclimate-flake8, then you can get the same image without CodeClimate-related code by using the megabytelabs/flake8 image. For a full listing of images to choose from, check out the DockerHub page for this project.

Note: The DockerHub page for the CodeClimate engine version of this project is located on the aforementioned DockerHub page and also on a separate DockerHub page prefixed with codeclimate-.

Slim Build

All of our CodeClimate engine projects include two variants. One is the regular build that is created using Docker and the Dockerfile that is in the root of this project. The other build is a slim build that is built via DockerSlim. If you are looking for stability and want to take advantage of the slim build, then you should specify an exact version when referencing the image (e.g. megabytelabs/codeclimate-package:5.4.0-slim instead of megabytelabs/codeclimate-package:slim).

Versioning

Whenever possible, we tag our Docker image versions with the corresponding version of the software that the Docker image is used for. For instance, if you are using the ESLint CodeClimate image, then the megabytelabs/codeclimate-eslint:5.4.0 image means that it likely includes ESLint at version 5.4.0. There may be some exceptions but this is what we strive for.

Testing

Multiple methods of testing are used to ensure both the latest and slim build function properly. The Dockerfile-test.yml file in the root of the repository is a container-structure-test that ensures that each container is working properly. On top of that, we also run commands on sample projects stored in test/compare/ to ensure that the output from the latest image matches the output from the slim image. In some other scenarios, we also include unit tests for custom code written for the CodeClimate engine.

Requirements

Docker

Development Requirements

DockerSlim - Used for generating compact, secure images
jq - Used for interacting with JSON
Node.js (>=14.18) - Utilized to add development features like a pre-commit hook and maintenance tasks
Many more requirements that are dynamically installed as they are needed by our Taskfile.yml via our custom go-task/task fork named Bodega

If you choose to utilize the development tools provided by this project then at some point you will have to run bash start.sh (or npm i which calls bash start.sh after it is done). The start.sh script will attempt to automatically install any requirements (without sudo) that are not already present on your build system to the user's ~/.local/bin folder. The start.sh script also takes care of other tasks such as generating the documentation by calling tasks defined in the Taskfile.yml. For more details on how the optional requirements are used and set up, check out the CONTRIBUTING.md guide.

When you are ready to start development, run task --menu to open an interactive dialog that will help you understand what build commands we have already engineered for you.

Examples

There are several different ways you can use the Docker container provided by this project. For starters, you can test the tool out locally by running:

docker run -it -v ${PWD}:/work -w /work --rm megabytelabs/flake8:latest --help

This allows you to run Flake8 without installing it locally. It also removes the image from your system when you are done. This could be good for security since the application is within a container and also keeps your file system clean.

You can also add a bash alias to your ~/.bashrc file so that you can run the Flake8 command at any time. To do this, add the following snippet to your ~/.profile file (or equivalent):

flake8() {
  docker run -it -v ${PWD}:/work -w /work --rm megabytelabs/flake8:latest "$@"
}

Note: Some CLI tools run without any arguments passed in. For example, the CLI tool ansible-lint runs by simply entering ansible-lint in the terminal. Our Docker images default command is to show the version so to get around this quirk you would run ansible-lint ..

Integrating with GitLab CI

The main purpose of this project is to build a Docker container that can be used in CI pipelines and during GitLab CI CodeClimate analysis.

CodeClimate GitLab CI Example

If you are interested in improving the GitLab web UI for your merge requests, then you can take a peek at our GitLab CI configuration for CodeClimate. It has to run in a single CI stage that includes all the linters you want to report on because GitLab CI only accepts one CodeClimate report artifact.

CodeClimate CLI Walkthrough

If you want to incorporate CodeClimate into your project, you need to ensure that you have a .codeclimate.yml properly setup in the root of your project. At the very minimum, the file might look something like this:

---
engines:
  flake8:
    enabled: true
exclude_paths:
  - 'node_modules/**'

Also, before you run the CodeClimate CLI, you need to ensure that this project's CodeClimate image is pulled to your local cache and properly tagged for CodeClimate. You can accomplish this by running:

docker pull megabytelabs/codeclimate-flake8:latest
docker tag megabytelabs/codeclimate-flake8:latest codeclimate/codeclimate-flake8:latest

After that, you need to invoke the CodeClimate CLI by passing the --dev parameter. This may seem hacky but it is the only way of using CodeClimate engines that are not officially hosted in the codeclimate namespace on DockerHub. Your CLI command might look something like this:

brew install codeclimate/formulae/codeclimate
codeclimate analyze --dev

Or if you want to see an HTML report:

brew install codeclimate/formulae/codeclimate
codeclimate analyze --dev -f html > codeclimate-report.html

Standalone Integration

If you want to incorporate this CI pipeline tool into GitLab CI project without CodeClimate integration then your first step would be to create a .gitlab-ci.yml file in the root of your GitLab repository. Your .gitlab-ci.yml file should look something like this:

---
stages:
  - lint

include:
  - remote: https://gitlab.com/megabyte-labs/gitlab-ci/-/raw/master/lint/flake8.gitlab-ci.yml

That is it! Flake8 will now run anytime you commit code (that matches the parameters laid out in the remote: file above). Ideally, for production, you should copy the source code from the remote: link above to another location and update the remote: link to the file's new location. That way, you do not have to worry about any changes that are made to the remote: file by our team.

Configuring Flake8

This project is configured to look for the configuration for Flake8 at .config/flake8.toml first before reverting to the standard location.

Extending Flake8

This engine currently just loads Flake8 without any additional plugins. If you feel like this engine could be more valuable with additional plugins, and you know how to properly implement the plugins, please open a PR.

Building the Docker Container

You may have a use case that requires some modifications to our Docker image. After you make changes to the Dockerfile, you can upload your custom container to Docker Hub using the following code:

export DOCKERHUB_USERNAME=Your_DockerHub_Username_Here
export DOCKERHUB_PASSWORD=Your_DockerHub_Password_Here
docker login -u "$DOCKERHUB_USERNAME" -p "$DOCKERHUB_PASSWORD" docker.io
docker build --pull -t "$DOCKERHUB_USERNAME/flake8:latest" .
docker push "$DOCKERHUB_USERNAME/flake8:latest"

After setting your DockerHub username and password, the commands above will build the Docker image and upload it to Docker Hub where it will be publicly accessible. You can see this logic being implemented as a GitLab CI task here. This GitLab CI task works in conjunction with the .gitlab-ci.yml file in the root of this repository.

Building a Slim Container

Some of our repositories support creating a slim build via DockerSlim. According to DockerSlim's GitHub page, slimming down containers reduces the final image size and improves the security of the image by reducing the attack surface. It makes sense to create a slim build for anything that supports it, including Alpine images. On their GitHub page, they report that some images can be reduced in size by up to 448.76X. This means that if your image is naturally 700MB then it can be reduced to 1.56MB! It works by removing everything that is unnecessary in the container image.

As a convenience feature, we include a command defined in package.json that should build the slim image. Just run task docker:build after running npm i (or bash start.sh if you do not have Node.js installed) in the root of this repository to build both the latest and slim builds.

To build and publish a slim Dockerfile to Docker Hub manually, you can use the following as a starting point:

export DOCKERHUB_USERNAME=Your_DockerHub_Username_Here
export DOCKERHUB_PASSWORD=Your_DockerHub_Password_Here
docker login -u "$DOCKERHUB_USERNAME" -p "$DOCKERHUB_PASSWORD" docker.io
docker build -t "$DOCKERHUB_USERNAME/flake8:latest" .
docker-slim build --tag $DOCKERHUB_USERNAME/flake8:slim {{#if (eq (typeOf dockerSlimCommand) "string")}}dockerSlimCommand}}{{/if{{#if (not (eq (typeOf dockerSlimCommand) "string"))}}dockerSlimCommand[slug]}}{{/if $DOCKERHUB_USERNAME/flake8:latest
docker push "$DOCKERHUB_USERNAME/flake8:slim"

It may be possible to modify the DockerSlim command above to fix an issue or reduce the footprint even more than our command. You can modify the slim build command inline in the package.json file under blueprint.dockerSlimCommand. Some of our repositories have multiple build targets in the Dockerfile so those repositories will have multiple dockerSlimCommands.

If you come up with an improvement, please do open a pull request. And again, make sure you replace DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD in the snippet above with your Docker Hub username and password. The commands in the snippet above will build the slim Docker image and upload it to Docker Hub where it will be publicly accessible.

Build Tools

You might notice that we have a lot of extra files considering that this repository basically boils down to a single Dockerfile. These extra files are meant to make team development easier, predictable, and enjoyable. If you have a recent version of Node.js installed, you can get started using our build tools by running npm i (or by running bash start.sh if you do not currently have Node.js installed) in the root of this repository. After that, you can run task --list to see a list of the available development features. Alternatively, you can run task --menu to view an interactive menu that will guide you through the development process.

Note: We use a custom-built version of go-task/task so if you already have it installed then you should either replace it with our version or use a different bin name for task.

For more details, check out the CONTRIBUTING.md file.

Philosophy

Utilizing Continuous Integration (CI) tools can improve developer efficiency drastically. They allow you to do things like scan new code for possible errors and automatically deploy new software.

This repository is home to the build instructions for a Docker container that is just one piece to the CI puzzle. Nearly all of our CI pipeline Docker projects serve a single purpose.

Instead of using one of the countless pretty_name public Docker images available, we create it in-house so we know exactly what code is present in the container. We also ensure that all of our CI pipeline images are as small as possible so that our CI environment can download and run the specific task as quickly as possible. Using this repository as a base, you too can easily create your own in-house CI pipeline container image.

At first glance, you might notice that there are many files in this repository. Nearly all the files and folders that have a period prepended to them are development configurations. The tools that these files and folders configure are meant to make development easier and faster. They are also meant to improve team development by forcing developers to follow strict standards so that the same design patterns are used across all of our repositories.

Contributing

Contributions, issues, and feature requests are welcome! Feel free to check the issues page. If you would like to contribute, please take a look at the contributing guide.

Brian Zalewski

Affiliates

Below you will find a list of services we leverage that offer special incentives for signing up for their services through our special links: