Backend Utilities

The backend utilities provide essential helper functions and transformers that support the core application functionality. These utilities handle common tasks such as authentication, data transformation, and array manipulation.

Overview

The utilities directory contains:

Copilot Admin Authorisation - Managing admin access and team list retrieval with caching
GitHub App Authentication - Secure authentication with GitHub APIs
Project Data Transformation - Converting between data formats
Technology Array Management - Updating technology arrays

Copilot Admin Authorisation

`copilotAdminChecker.js`

Manages Copilot admin authorisation and team list retrieval with in-memory caching for performance optimisation.

Method: `checkCopilotAdminStatus(userToken)`

Determines if a user has Copilot admin privileges and returns the appropriate team list.

Parameters:

userToken (string) - GitHub OAuth access token for the authenticated user

Returns: Promise resolving to an object:

{
  isAdmin: boolean,           // Whether user is a Copilot admin
  teams: Array<Object>,       // Teams the user can view
  userTeamSlugs: Array<string> // Team slugs the user is a member of
}

Authorisation Logic:

Fetches user's GitHub teams via GitHub API
Checks if user belongs to any teams in admin_teams.json (from S3)
If admin: Returns teams from teams_history.json (cached for 1 hour)
If not admin: Returns only user's personal teams

Caching Behaviour:

Team list is cached in memory for 1 hour (configurable via TEAMS_CACHE_TTL)
First request downloads teams_history.json from S3
Subsequent requests return instantly from cache
Cache automatically expires after TTL
Cache resets on server restart

Example:

const { checkCopilotAdminStatus } = require('./utilities/copilotAdminChecker');

async function getAvailableTeams(req, res) {
  const userToken = req.cookies.githubUserToken;

  try {
    const result = await checkCopilotAdminStatus(userToken);

    if (result.isAdmin) {
      console.log(`Admin viewing ${result.teams.length} teams`);
    } else {
      console.log(`User viewing ${result.teams.length} personal teams`);
    }

    res.json(result);
  } catch (error) {
    console.error('Authorisation check failed:', error);
    res.status(500).json({ error: 'Authorisation failed' });
  }
}

Configuration:

COPILOT_BUCKET_NAME - S3 bucket containing team data (default: "sdp-dev-copilot-usage-dashboard")
TEAMS_CACHE_TTL - Cache duration in milliseconds

GitHub App Authentication

`getAppAndInstallation.js`

Provides secure authentication for GitHub API operations using GitHub App credentials stored in AWS Secrets Manager.

Configuration

The utility requires the following environment variables:

GITHUB_ORG - Target GitHub organisation (default: "ONSdigital")
GITHUB_APP_ID - GitHub App ID for authentication
AWS_SECRET_NAME - AWS Secrets Manager secret containing the private key
AWS_REGION - AWS region for Secrets Manager (configured as "eu-west-2")

Method: `getAppAndInstallation()`

Authenticates with GitHub and returns an installation-scoped Octokit instance.

Returns: Promise resolving to authenticated Octokit instance

Process:

Retrieves GitHub App private key from AWS Secrets Manager
Creates GitHub App instance using App ID and private key
Gets installation details for the specified organisation
Returns installation-scoped Octokit client

Example:

const { getAppAndInstallation } = require('./utilities/getAppAndInstallation');

async function getRepositories() {
  try {
    const octokit = await getAppAndInstallation();
    const response = await octokit.request('GET /orgs/{org}/repos', {
      org: 'ONSdigital',
    });
    return response.data;
  } catch (error) {
    console.error('Authentication failed:', error);
  }
}

Project Data Transformation

`projectDataTransformer.js`

Transforms project data from raw JSON format to structured CSV format for export and analysis.

Method: `transformProjectToCSVFormat(project)`

Converts a project object from JSON structure to CSV-compatible format.

Parameters:

project (object) - Raw project data object

Returns: Transformed project object in CSV format

Key Transformations:

User Role Extraction

// Technical contact with ONS email
const technicalContactUser = project.user.find(
  (u) =>
    u.roles.includes('Technical Contact') &&
    (u.email?.includes('@ons.gov.uk') || u.email?.includes('@ext.ons.gov.uk'))
);

// Format: email (grade)
const technicalContact = technicalContactUser
  ? `${technicalContactUser.email} (${technicalContactUser.grade})`
  : '';

Technology Array Formatting

const mainLanguages = project.architecture.languages.main.join('; ');
const frameworks = project.architecture.frameworks.others.join('; ');

Development Context

const developed =
  project.developed[1] != ''
    ? `${project.developed[0]} with ${project.developed.slice(1).join(', ')}`
    : project.developed[0];

Output Structure:

{
  Project: string,                    // Project name
  Project_Short: string,              // Short project name
  Programme: string,                  // Programme name
  Programme_Short: string,            // Short programme name
  Description: string,                // Project description
  Stage: string,                      // Development stage
  Developed: string,                  // Development context
  Technical_Contact: string,          // Technical contact (email + grade)
  Delivery_Manager: string,           // Delivery manager (email + grade)
  Language_Main: string,              // Main programming languages
  Language_Others: string,            // Other languages
  Language_Frameworks: string,        // Frameworks used
  Hosted: string,                     // Hosting platforms
  Architectures: string,              // Architecture details
  Source_Control: string,             // Source control system
  Repo: string,                       // Repository URLs
  CICD: string,                       // CI/CD tools
  Datastores: string,                 // Data storage solutions
  Database_Technologies: string,      // Database technologies
  Project_Tools: string,              // Project tracking tools
  Documentation: string,              // Documentation links
  Infrastructure: string,             // Infrastructure tools
  Code_Editors: string,               // Code editors used
  Communication: string,              // Communication tools
  Collaboration: string,              // Collaboration tools
  Incident_Management: string,        // Incident management tools
  Documentation_Tools: string,        // Documentation tools
  UI_Tools: string,                   // UI/UX tools
  Diagram_Tools: string               // Diagramming tools
}

Example Usage:

const { transformProjectToCSVFormat } = require('./utilities/projectDataTransformer');

const rawProject = {
  details: [{ name: 'Digital Platform', short_name: 'DP' }],
  user: [
    {
      email: 'tech.lead@ons.gov.uk',
      grade: 'Senior Developer',
      roles: ['Technical Contact'],
    },
  ],
  architecture: {
    languages: {
      main: ['JavaScript', 'Python'],
      others: ['Go'],
    },
  },
  // ... other project data
};

const csvProject = transformProjectToCSVFormat(rawProject);
console.log(csvProject.Technical_Contact); // "tech.lead@ons.gov.uk (Senior Developer)"

Technology Array Management

`updateTechnologyInArray.js`

Provides helper functionality for updating technology names within arrays, supporting technology normalisation processes.

Method: `updateTechnologyInArray(array, from, to)`

Updates technology names in arrays while tracking whether changes occurred.

Parameters:

array (string[]) - Array of technology names
from (string) - Original technology name to replace
to (string) - New technology name

Returns: Object containing updated array and change status

{
  array: string[],    // Updated array
  updated: boolean    // Whether an update occurred
}

Features:

Case-sensitive matching
Non-destructive operation (returns new array)
Change tracking for audit purposes
Handles null/undefined arrays gracefully

Example Usage:

const { updateTechnologyInArray } = require('./utilities/updateTechnologyInArray');

const technologies = ['React', 'Vue.js', 'Angular'];

// Update Vue.js to Vue
const result = updateTechnologyInArray(technologies, 'Vue.js', 'Vue');
console.log(result.array); // ["React", "Vue", "Angular"]
console.log(result.updated); // true

// Attempt to update non-existent technology
const noChange = updateTechnologyInArray(technologies, 'Svelte', 'SvelteKit');
console.log(noChange.updated); // false

Integration Examples

Complete Authentication Flow

const { getAppAndInstallation } = require('./utilities/getAppAndInstallation');

async function authenticatedGitHubOperation() {
  try {
    // Get authenticated client
    const octokit = await getAppAndInstallation();

    // Perform GitHub API operations
    const repos = await octokit.request('GET /orgs/{org}/repos', {
      org: process.env.GITHUB_ORG,
    });

    return repos.data;
  } catch (error) {
    console.error('GitHub operation failed:', error);
    throw error;
  }
}

Bulk Data Transformation

const { transformProjectToCSVFormat } = require('./utilities/projectDataTransformer');

async function exportProjectsToCSV(projects) {
  const csvData = projects
    .map((project) => {
      try {
        return transformProjectToCSVFormat(project);
      } catch (error) {
        console.error(`Failed to transform project ${project.details?.[0]?.name}:`, error);
        return null;
      }
    })
    .filter(Boolean);

  return csvData;
}

Technology Normalisation

const { updateTechnologyInArray } = require('./utilities/updateTechnologyInArray');

function normaliseProjectTechnologies(project, normalisationMap) {
  let updated = false;
  const updatedProject = { ...project };

  // Update main languages
  for (const [from, to] of Object.entries(normalisationMap)) {
    const result = updateTechnologyInArray(project.architecture.languages.main, from, to);

    if (result.updated) {
      updatedProject.architecture.languages.main = result.array;
      updated = true;
    }
  }

  return { project: updatedProject, updated };
}

Implementation Notes

All utilities follow consistent error handling patterns
Functions are designed to be composable and reusable
Authentication utilities handle AWS/GitHub integration securely
Data transformers preserve original data structure integrity
Array utilities provide audit trails for changes
Compatible with both synchronous and asynchronous workflows

Backend Utilities

Overview

Copilot Admin Authorisation

copilotAdminChecker.js

Method: checkCopilotAdminStatus(userToken)

GitHub App Authentication

getAppAndInstallation.js

Configuration

Method: getAppAndInstallation()

Project Data Transformation

projectDataTransformer.js

Method: transformProjectToCSVFormat(project)

User Role Extraction

Technology Array Formatting

Development Context

Technology Array Management

updateTechnologyInArray.js

Method: updateTechnologyInArray(array, from, to)

Integration Examples

Complete Authentication Flow

Bulk Data Transformation

Technology Normalisation

Implementation Notes

`copilotAdminChecker.js`

Method: `checkCopilotAdminStatus(userToken)`

`getAppAndInstallation.js`

Method: `getAppAndInstallation()`

`projectDataTransformer.js`

Method: `transformProjectToCSVFormat(project)`

`updateTechnologyInArray.js`

Method: `updateTechnologyInArray(array, from, to)`