Implementing CI/CD in Catalyst
Table of Contents:
- Introduction to Continuous Integration and Continuous Deployment
- Use Case
Continuous Integration (CI) is the development practice of integrating new or changed code into an existing code repository automatically. The configured CI system then verifies each integration by performing immediate automated builds and tests of the application every time the code is updated and saved. The CI system automates and streamlines the entire process, allowing you to detect and rectify errors in the code easily, and enabling you to identify specific changes in the code that caused errors.
Continuous Deployment (CD) is the approach of automatically deploying the code that was built and tested by the CI system, to a production system or a staging system. Continuous Deployment occurs after the Continuous Integration stage, and it ensures that the updated code is always ready to be used.
A well-defined deployment pipeline is crucial to ensure the CI/CD approach is completely effective. The deployment pipeline dictates the logic and the steps involved in implementing CI/CD, from the initial testing stages to the final deployment of the code to production. Once you configure the deployment pipeline for your application and define the workflow, you can enable complete automation in integrating and deploying your application code.
Together, the CI/CD practice ensures a significant reduction in the cost and time of an application building process, and decreases the manual workload involved. Whenever you make changes to your code, no matter how minor, CI/CD enables immediate automated testing on the code to ensure that the change fits well with the existing codebase.
If you update your application constantly and perform unit tests on it every time, it is a good practice to push the code to a staging environment where you can perform integrated tests in a production-like environment. If your code is especially reliant on external dependencies and services, it becomes harder to ensure the integrity of the code maintained for every minor update. Instant deployment and thorough testing of the code is therefore highly important to reduce the risks and increase the production speed in the software development process.
CI/CD is now considered to be among the best practices for software development as it ensures the rapid delivery of reliable, risk-free, high-quality code.
Catalyst provides you with the ability to implement CI/CD in the applications and microservices that you build using it. You can integrate Catalyst with external services and enable a consistent data flow between them. You can automate code testing and application builds, access and work on your code remotely, and deploy the tested code to Catalyst using CI/CD.
The benefits of implementing CI/CD in a Catalyst application can be summarized as follows:
- Faster and more efficient application development process as a result of automation
- Enhanced security because of the frequent testing done on the application code
- Early detection of problematic areas leading to quicker and easier resolution
- Reduction in overhead in the application development process
- Reduction in manual work involved in testing and deploying code
- Encourages smaller incremental changes while developing new features
- Enables communication, collaboration, and code between team members
The aim of this help guide is to educate the learner about implementing CI/CD in a Catalyst application with other external dependent services. We will discuss this with a specific use case as an example.
The Cat.ly application, that we will discuss about in this help guide, is a simple URL shortener application that reduces the length of URLs for the convenience in sharing and accessing them. The Cat.ly application is built using Catalyst as a serverless platform and implements the CI/CD approach, involving external integrations.
Access the full length tutorial on building the Cat.ly application here.
The Cat.ly application contains two fundamental components of Catalyst: Functions and Client. The Advanced I/O function used in Cat.ly contains the API that will shorten the URL that the user provides. The client component is the frontend of the application that can be hosted in Catalyst through web client hosting.
The Cat.ly application contains the following directory structure:
You can learn about the Catalyst project directory from the Project Directory Structure help page.
The development environment of the Cat.ly application involves the following services:
- Catalyst CLI and Web Console
The Catalyst CLI and web console are used to initialize, develop, and deploy the function and client components. Catalyst CLI will be remotely accessed by external services to perform operations required to enable CI/CD.
The React framework is used to build the user framework of the Cat.ly application. It is created using the create-react-app node module to generate a boilerplate version of a React application.
A GitHub repository is used to host the code of the Cat.ly application and to enable the integration of its code with the CI system. The repository is linked with Catalyst to enable easy transfer and synchronization of the application's code.
Travis CI is used to automatically build and test the code of the Cat.ly application. A travis.yml file is created to define the build and test specifications of the Cat.ly application. This file is added in the GitHub repository containing the application's code, which is then linked to Travis CI.
Before you learn the steps for implementing CI/CD in the Cat.ly application, you will need to understand some of the key concepts involved in detail.
Travis CI is a continuous integration and deployment service that is used to test and build software based on the steps you define. Travis CI is associated with GitHub and it can perform tests on the code hosted in your GitHub repositories. When you sign up for an account in Travis CI, you must sign up with your GitHub account.
You can then activate the GitHub repository that contains the code to be tested and deployed using Travis CI.
This will enable Travis CI to import the files from GitHub and perform continuous integration on them every time new commits are pushed or pull requests are made in the repository or in a specific branch.
You can define the flow of the desired building and testing environment and configure Travis CI by including a travis.yml file in the root folder of your application. You can install the required dependencies for the components of the application, such as node modules, through the travis.yml file.
Travis CI runs the commands defined in travis.yml whenever any files are updated in the GitHub's repository or branch. When the automated testing and building process is complete, you will be notified of the results in the way you have specified. If Travis CI encounters bugs or errors in any of the steps in the process, such as missing dependencies or issues with the configuration files, it will alert you about them. You can learn more from the Travis CI documentation.
The travis.yml file of the Cat.ly application contains the following code:
language: node_js node_js: - 8 install: - npm install -g zcatalyst-cli jobs: include: - stage: functions before_script: - cd functions/catly/server script: - echo 'function!!' - echo 'Installing Deps!' - npm install after_success: - echo 'deploying function!' - catalyst deploy --only functions --project 700000000045014 --verbose - stage: client before_script: - cd client/app script: - echo 'client!!' - echo 'Installing Deps!' - yarn install - yarn build - cp ../client-package.json build/. - cd build after_success: - echo 'deploying client!' - catalyst deploy --only client --project 700000000045014 --verbose
Let's take a closer look at the content in the travis.yml file.
This YAML text file dictates the terms of the tasks to be performed in various stages. We can break the execution of the test specifications down in the following way:
- The first two lines of the YAML file specify the programming language and version of the configuration. In this case, it is Node.js v8.0.
- The install parameter specifies the npm command for installing Catalyst CLI as a global module. You can learn more about the installation steps from the CLI documentation.
- The tasks to be performed are defined under the jobs parameter. There are essentially two tasks to be performed: deploying the function and deploying the client. These are defined in two different stages. These stages work independently of each other, so even if the execution of one stage fails, the other can run and be successful.
Deploying the Advanced I/O function:
Deploying the Advanced I/O function of the Cat.ly application is straightforward. As discussed earlier, the function contains the API that will shorten the URL that the user provides. You can check the function's directory structure from this section.
The process of installing the node modules and dependencies, and deploying the function, is divided into three parts: before_script, script, and after_success in the function stage. You can learn more about scripts from the Catalyst Scripts help page.
- The before_script lists the task to be performed before executing the catalyst run-script command: navigating to the server folder inside the functions directory that contains the function's code.
- The script indicates the action to be executed while running the script: installing the npm package and dependencies inside the server folder.
- The after_success specifies the command to be executed after the success of the script execution: deploy the function specifying the project ID in the debug (--verbose) mode.
Deploying the client:
The next stage in the process is to deploy the client component. This is also divided into the same three parts: before_script, script, and after_success.
The client component of the Cat.ly application is created as a React app. As with all React apps, the client component must be compiled into a production-ready build before being deployed.
- The before_script parameter informs Travis CI to navigate to the client directory.
- The script parameter contains commands to install dependencies using Yarn, and then build the client component by running the yarn build command which is similar to the npm build command. This creates the build folder in the root directory of the client, and the application files are added in it.
Since the client component is built from scratch using Yarn, we must add the client-package.json file inside the build folder manually. The next command copies the client-package.json file from the client folder to the build folder. The build folder is now the root folder of the client component, containing all the same application files, configuration files, and dependencies as the original client folder. Therefore, we must navigate to the build folder before deploying the client component.
- The after_success specifies the catalyst deploy command to be executed to deploy the client component. We specify the project ID of the project and execute the command in the debug (--verbose) mode.
Note: The catalyst.json configuration file must specify the source of the client component as the build folder, and not the client folder, to prevent unnecessary files from being deployed. This information is included in the file before it is hosted in the GitHub repository.
Another important feature of implementing CI/CD with Travis CI, is using Catalyst tokens to validate the CLI sessions. The Catalyst CLI tokens allow you to perform CLI activities from the remote system without requiring an active log-in session in the CLI. The token is used by Travis while executing the CLI commands specified in the travis.yml file. Catalyst does not allow a remote system to access it without a valid token. The token therefore acts as an authenticating factor for a particular user account and allows you to map your command executions to the active project. You can learn about generating a CLI token from the Working with Tokens help page.
We also specify the project ID of the Catalyst project that is associated with the Cat.ly application in the travis.yml file.
You can generally use the CLI token by specifying it as an option with the CLI command. This is done by typing "--token" and the token's value following the CLI command. However, we have not used the --token option in the travis.yml file after the CLI commands, because Travis CI enables you to add the CLI token as an environment variable.
You can add the CLI token as an environment variable by specifying the name as 'CATALYST_TOKEN' and entering the token under value in your repository's settings in Travis.
Adding the token as an environment variable ensures complete security of access to the Catalyst CLI. It prevents you from openly disclosing the token in the travis.yml file stored in the GitHub repository or in the Travis CI. Travis will automatically use this token for authorization whenever it executes a CLI command.
Now that we have discussed Travis CI and Catalyst CLI tokens in detail, let us understand the steps involved in implementing CI/CD with Catalyst.
The general outline of developing the Cat.ly application and implementing CI/CD in it can be broadly summarized in the following steps:
- Creating and initializing a Catalyst project and its components using the CLI and coding the application.
- Setting up the travis.yml file containing the build and test specifications and including it in the project directory.
- Creating a GitHub repository to host the necessary application files and integrating it with Catalyst.
- Signing up for Travis using the GitHub account and linking the GitHub repository containing the code of Cat.ly application with Travis.
- Testing the CI/CD setup by committing and pushing the code in GitHub to trigger a Travis CI build.
- Verifying the success or failure of the Travis CI build from the repository's build status page.
If the build is successful and the CLI commands specified in the travis.yml file executed successfully, the deployed Cat.ly application will be hosted and available in the Catalyst web console.
With this CI/CD system set up, every time we make changes in the Cat.ly application's code and update it in the GitHub repository, the code will be automatically tested and deployed to Catalyst again by Travis CI.