Release Process
This document describes the TrainLoop Evals release process, including version bumping, changelog management, and publication workflows.
Overviewโ
TrainLoop Evals uses a semi-automated release process that combines manual release note creation with automated version bumping, building, and publishing.
Release Workflowโ
Step 1: Create Release Notesโ
Before bumping the version, create a release notes file in the releases/
directory. The file must start with a Summary:
line that briefly describes the release.
# For example, if releasing version 0.5.0, create releases/0.5.0.md
cat > releases/0.5.0.md << EOF
Summary: Major SDK refactoring with comprehensive testing infrastructure
**Major SDK Refactoring and Testing Infrastructure**
The changes in this release include:
### ๐ SDK Restructuring
- Renamed SDK from "evals-sdk" to "sdk"
- Renamed packages from "trainloop_evals" to "trainloop_llm_logging"
- Improved config functions to handle environment variable fallbacks
### ๐งช Testing Infrastructure
- Added comprehensive test suites with over 3000 lines of tests
- Implemented pytest configuration for Python SDK
- Implemented Jest configuration for TypeScript SDK
### ๐ Bug Fixes
- Fixed UI database initialization to handle empty directories gracefully
- Fixed concurrency issues in DuckDB
EOF
Step 2: Run the Release Scriptโ
Automated (recommended):
# Use the automated release script:
npm run release minor "Brief description of the release"
Manual bump (advanced users):
npm run bump minor
Step 3: GitHub Actions Handles Publishingโ
When merged to the main branch, GitHub Actions automatically:
- Creates a git tag
- Builds all packages (CLI, SDKs, Studio)
- Publishes to package repositories (PyPI, NPM)
- Updates deployment infrastructure
Version Bump Scriptโ
The version bump script performs the following operations:
-
Updates all package files with the new version:
package.json
files (root and components)pyproject.toml
files (Python packages)- Lock files (
package-lock.json
,poetry.lock
)
-
Updates CHANGELOG.md with a link to the release notes
-
Creates a git commit with the version changes
-
Pushes to the current branch (no tag creation - handled by GitHub Actions)
Release File Formatโ
Release files must follow this format:
Summary: Brief description of the release
[Rest of your detailed release notes in markdown format]
### ๐ New Features
- Feature 1
- Feature 2
### ๐ Bug Fixes
- Fix 1
- Fix 2
### ๐ Documentation
- Documentation improvements
### ๐ง Internal
- Internal changes
Important Requirements:
- The file must start with
Summary: <message>
- The summary is used for git commit messages and tags
- Release files should be markdown formatted for readability
- Use emojis and clear categorization for better readability
Changelog Outputโ
The changelog will contain links to the full release notes:
# Changelog
## 0.5.0 (2025-05-27)
[Release Notes](releases/0.5.0.md)
## 0.4.0 (2025-05-22)
โข Public Release
Build and Publishingโ
Automated Publishing (Production)โ
GitHub Actions handles all publishing when code is merged to main:
# .github/workflows/release.yml
- name: Build all packages
run: npm run build
- name: Publish CLI to PyPI
run: npm run publish:cli
- name: Publish SDKs to PyPI and NPM
run: npm run publish:sdk
- name: Publish Studio to NPM
run: npm run publish:studio
Manual Publishing (Development)โ
For testing releases:
# Build all components
npm run build
# Publish individual components
npm run publish:cli # CLI to PyPI
npm run publish:sdk # SDKs to PyPI and NPM
npm run publish:studio # Studio to NPM
# Or publish everything
npm run publish
Version Managementโ
Semantic Versioningโ
TrainLoop Evals follows semantic versioning:
- MAJOR version: Breaking changes
- MINOR version: New features, backwards compatible
- PATCH version: Bug fixes, backwards compatible
Monorepo Version Synchronizationโ
All packages in the monorepo share the same version number:
- CLI (
trainloop-cli
) - Python SDK (
trainloop-llm-logging
) - TypeScript SDK (
trainloop-llm-logging
) - Go SDK (
trainloop-llm-logging
) - Studio UI (
trainloop-studio
)
This ensures compatibility across all components.
Testing Releasesโ
Pre-release Testingโ
Before creating a release:
# Run all tests
npm test
# Build all packages
npm run build
# Test CLI locally
cd cli && poetry run trainloop --help
# Test Studio locally
cd ui && npm run dev
Post-release Verificationโ
After publishing:
# Test CLI installation
pipx install trainloop-cli
trainloop --version
# Test SDK installation
pip install trainloop-llm-logging
npm install trainloop-llm-logging
Rollback Processโ
If a release needs to be rolled back:
- Identify the issue and create a hotfix
- Create a patch release with the fix
- Update package repositories to mark the problematic version as deprecated
- Communicate the issue and resolution to users
Infrastructure Updatesโ
Release deployments may also update infrastructure:
# Update Pulumi infrastructure after release
npm run pulumi:bump
This ensures that any infrastructure changes required for the new version are deployed.
Release Checklistโ
- Create release notes file in
releases/
- Release notes start with
Summary:
line - Run
npm run release [major|minor|patch]
- Verify all tests pass
- Create pull request
- Merge to main branch
- Verify GitHub Actions deployment succeeds
- Test published packages
- Announce release in appropriate channels
Common Issuesโ
Build Failuresโ
If the build fails during release:
- Fix the issue in a separate commit
- Re-run the release process
- Ensure all package.json files are updated consistently
Publishing Failuresโ
If publishing fails:
- Check API tokens for PyPI and NPM
- Verify package names and versions
- Check for network connectivity issues
- Retry publishing individual components
Version Conflictsโ
If version conflicts occur:
- Ensure all package files have consistent versions
- Run
npm run bump
again to fix inconsistencies - Check for uncommitted changes that might interfere
Supportโ
For questions about the release process:
- Check the development documentation
- Open an issue on GitHub
- Contact the maintainers via GitHub discussions