The Script-First Approach to CI/CD: Building Portable and Testable Automation Pipelines

Table of Contents

When building CI/CD pipelines, I follow a fundamental principle: the pipeline configuration (YAML/JSON) should be a thin orchestration layer that primarily calls scripts, while the actual logic lives in version-controlled, testable scripts.

This approach treats pipeline configuration files as mere schedulers and environment providers, not as the primary home for build logic.

Core Principles
#

1. Pipeline Files Are Just Orchestrators
#

# Good: Pipeline as a simple orchestrator
name: Deploy Application
steps:
  - name: Build
    run: ./scripts/build.sh
  - name: Test
    run: ./scripts/test.sh
  - name: Deploy
    run: ./scripts/deploy.sh ${{ env.ENVIRONMENT }}

2. Scripts Contain the Logic
#

All meaningful logic should live in scripts that are:

Version controlled alongside your code
Executable locally for testing
Written in portable languages (bash, PowerShell, Python)
Self-documenting with clear naming and comments

3. Environment Variables as the Interface
#

Scripts should accept configuration through environment variables and parameters, making them platform-agnostic:

#!/bin/bash
# scripts/deploy.sh
ENVIRONMENT=${1:-staging}
REGION=${AWS_REGION:-us-east-1}
APP_VERSION=${VERSION:-$(git describe --tags --always)}

echo "Deploying version $APP_VERSION to $ENVIRONMENT in $REGION"
# Actual deployment logic here

Benefits of This Approach
#

1. Local Testing and Development
#

Developers can run and debug the exact same scripts locally:

# Test the build process locally before committing
./scripts/build.sh
./scripts/test.sh
ENVIRONMENT=dev ./scripts/deploy.sh

2. Platform Portability
#

Migrating between CI/CD platforms becomes trivial:

GitHub Actions:

- name: Deploy
  run: ./scripts/deploy.sh production

Azure DevOps:

- script: ./scripts/deploy.sh production
  displayName: 'Deploy to Production'

Jenkins:

stage('Deploy') {
    sh './scripts/deploy.sh production'
}

GitLab CI:

deploy:
  script:
    - ./scripts/deploy.sh production

3. Reduced Vendor Lock-in
#

Your automation logic isn’t tied to GitHub Actions expressions, Azure DevOps tasks, or Jenkins plugins. It’s just scripts that run anywhere.

4. Version Control and Code Review
#

Scripts are first-class citizens in your repository:

Track changes through git history
Review logic changes in pull requests
Apply the same coding standards as application code
Write tests for complex scripts

5. Reusability Across Projects
#

Well-written scripts can be shared across projects:

# Clone shared scripts repository
git clone https://github.com/yourorg/ci-scripts.git .ci-scripts
# Use them in your pipeline
.ci-scripts/standard-build.sh

Practical Implementation
#

Directory Structure
#

project/
├── .github/workflows/     # Or .azure-pipelines/, .gitlab-ci.yml, etc.
│   └── main.yml           # Minimal orchestration
├── scripts/               # All automation logic
│   ├── build.sh
│   ├── test.sh
│   ├── deploy.sh
│   ├── rollback.sh
│   └── utilities/         # Shared functions
│       └── common.sh
└── src/                   # Application code

Example: Complete Build Script
#

#!/bin/bash
# scripts/build.sh
set -euo pipefail

# Source common utilities
source "$(dirname "$0")/utilities/common.sh"

# Configuration
BUILD_OUTPUT="${BUILD_OUTPUT:-./dist}"
BUILD_CONFIG="${BUILD_CONFIG:-Release}"

log_info "Starting build process..."
log_info "Configuration: $BUILD_CONFIG"
log_info "Output directory: $BUILD_OUTPUT"

# Clean previous builds
rm -rf "$BUILD_OUTPUT"
mkdir -p "$BUILD_OUTPUT"

# Restore dependencies
if [ -f "package.json" ]; then
    log_info "Installing npm dependencies..."
    npm ci
elif [ -f "requirements.txt" ]; then
    log_info "Installing Python dependencies..."
    pip install -r requirements.txt
elif [ -f "*.csproj" ]; then
    log_info "Restoring .NET dependencies..."
    dotnet restore
fi

# Build the application
if [ -f "package.json" ]; then
    npm run build
elif [ -f "setup.py" ]; then
    python setup.py build
elif [ -f "*.csproj" ]; then
    dotnet build -c "$BUILD_CONFIG" -o "$BUILD_OUTPUT"
fi

log_success "Build completed successfully!"

Example: Minimal Pipeline Configuration
#

# .github/workflows/main.yml
name: CI/CD Pipeline
on: [push, pull_request]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build
        run: ./scripts/build.sh

      - name: Test
        run: ./scripts/test.sh

      - name: Deploy
        if: github.ref == 'refs/heads/main'
        run: ./scripts/deploy.sh production
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Guidelines for AI Tools and Developers
#

When building automation pipelines following this approach:

1. Start with Scripts, Not Pipeline Config
#

Begin by writing the scripts that perform the actual work. Only then create the minimal pipeline configuration to orchestrate them.

2. Keep Pipeline-Specific Features to a Minimum
#

Avoid deep integration with platform-specific features:

❌ GitHub Actions marketplace actions for basic tasks
❌ Azure DevOps task groups for common operations
❌ Jenkins plugins for standard functionality
✅ Scripts that call standard CLI tools

3. Use Platform Features Only for What Scripts Can’t Do
#

Reserve platform-specific features for:

Secret management
Artifact storage between jobs
Matrix builds and parallelization
Environment provisioning

4. Make Scripts Defensive and Informative
#

# Check prerequisites
command -v docker >/dev/null 2>&1 || { echo "Docker is required but not installed."; exit 1; }

# Provide clear output
echo "================================================"
echo "Deployment Configuration:"
echo "  Environment: $ENVIRONMENT"
echo "  Region: $REGION"
echo "  Version: $VERSION"
echo "================================================"

5. Document Script Requirements
#

Include a header comment in each script:

#!/bin/bash
#
# build.sh - Build the application
#
# Requirements:
#   - Node.js 18+ or Python 3.8+ or .NET 6+
#   - Docker (optional, for container builds)
#
# Environment Variables:
#   - BUILD_CONFIG: Debug|Release (default: Release)
#   - BUILD_OUTPUT: Output directory (default: ./dist)
#
# Usage:
#   ./scripts/build.sh
#

Testing Strategy
#

Local Script Testing
#

Create a test harness for your scripts:

# scripts/test-scripts.sh
#!/bin/bash

# Test build script
echo "Testing build script..."
BUILD_OUTPUT=/tmp/test-build ./scripts/build.sh || exit 1

# Test deploy script in dry-run mode
echo "Testing deploy script..."
DRY_RUN=true ./scripts/deploy.sh staging || exit 1

echo "All script tests passed!"

Continuous Integration for Scripts
#

- name: Validate Scripts
  run: |
    # Check syntax
    find scripts/ -name "*.sh" -exec bash -n {} \;
    # Run shellcheck
    find scripts/ -name "*.sh" -exec shellcheck {} \;
    # Execute test harness
    ./scripts/test-scripts.sh

Migration Example
#

When moving from Jenkins to GitHub Actions, instead of rewriting everything:

Before (Jenkins-specific):

pipeline {
    stages {
        stage('Build') {
            steps {
                sh 'npm install'
                sh 'npm run build'
                archiveArtifacts artifacts: 'dist/**/*'
            }
        }
    }
}

After (Portable):

// Jenkins
pipeline {
    stages {
        stage('Build') {
            steps { sh './scripts/build.sh' }
        }
    }
}

# GitHub Actions
- run: ./scripts/build.sh

Conclusion
#

By adopting a script-first approach to CI/CD:

You own your automation logic, not the CI/CD platform
Developers can test pipelines locally without pushing commits
Migration between platforms becomes a configuration change, not a rewrite
Knowledge is transferable - learn scripting once, use everywhere

Remember: The best pipeline is one you can run on your laptop. If it works locally, it will work in CI/CD.

This approach has served me well across migrations between Jenkins, Azure DevOps, GitHub Actions, and GitLab CI. The scripts remain constant; only the thin orchestration layer changes.

Quick Reference for AI Assistants
#

When asked to create a CI/CD pipeline, follow these steps:

First, create script files in a scripts/ directory
Put all logic in these scripts (build, test, deploy, etc.)
Make scripts accept configuration via environment variables
Create minimal pipeline YAML that just calls these scripts
Ensure scripts can run both locally and in CI/CD environments
Add proper error handling and logging to scripts
Document script requirements and usage
Avoid platform-specific features unless absolutely necessary

This approach ensures maximum portability, testability, and maintainability of automation pipelines.

Core Principles#

1. Pipeline Files Are Just Orchestrators#

2. Scripts Contain the Logic#

3. Environment Variables as the Interface#

Benefits of This Approach#

1. Local Testing and Development#

2. Platform Portability#

3. Reduced Vendor Lock-in#

4. Version Control and Code Review#

5. Reusability Across Projects#

Practical Implementation#

Directory Structure#

Example: Complete Build Script#

Example: Minimal Pipeline Configuration#

Guidelines for AI Tools and Developers#

1. Start with Scripts, Not Pipeline Config#

2. Keep Pipeline-Specific Features to a Minimum#

3. Use Platform Features Only for What Scripts Can’t Do#

4. Make Scripts Defensive and Informative#

5. Document Script Requirements#

Testing Strategy#

Local Script Testing#

Continuous Integration for Scripts#

Migration Example#

Conclusion#

Quick Reference for AI Assistants#