Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions PACKAGE_EXCEPTIONS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Package Exceptions Configuration

This document explains how to configure package-specific exceptions for different platforms and architectures.

## Overview

The build system supports multiple PyPI index URLs and intelligent package handling:

- **Multiple Index URLs**: Support for additional PyPI repositories via command-line arguments
- **Smart Fallback**: Automatically tries binary wheels first, then falls back to source builds
- **Platform Exceptions**: Configure packages that need special handling for specific platforms

The build system now supports three types of package exceptions:

1. **Skip**: Packages that should be completely skipped for specific platforms
2. **Force Source**: Packages that should always be built from source for specific platforms
3. **Platform Specific**: Advanced configuration with custom actions and reasons

## Configuration File

Package exceptions are configured in `package-exceptions.json` at the root of the project.

## Extra Index URLs

You can specify additional PyPI index URLs when running the build script:

```bash
# Build with additional index URLs
node ./scripts/build.js 3.12.6 "https://custom.pypi.org/simple/,https://another.pypi.org/simple/"

# Build with single additional index
node ./scripts/build.js 3.12.6 "https://custom.pypi.org/simple/"
```

The build system will:
1. Always include the default NVIDIA PyPI index (`https://pypi.nvidia.com`)
2. Add any additional URLs you specify
3. Search all indexes when downloading packages

## Configuration Structure

```json
{
"skip": {
"package-name": {
"platform-key": "reason for skipping"
}
},
"forceSource": {
"package-name": {
"platform-key": "reason for forcing source build"
}
},
"platformSpecific": {
"package-name": {
"platform-key": {
"action": "skip|forceSource",
"reason": "detailed explanation"
}
}
}
}
```

## Platform Keys

Platform keys follow the format: `{os}-{arch}`

- `linux-x64` - Linux AMD64
- `linux-aarch64` - Linux ARM64
- `macosx-x64` - macOS Intel
- `macosx-aarch64` - macOS Apple Silicon
- `windows-x64` - Windows AMD64

## Examples

### Skipping CUDA Packages on Non-Linux Platforms

```json
{
"skip": {
"cudf-cu12": {
"macosx-x64": "CUDA packages not supported on macOS",
"macosx-aarch64": "CUDA packages not supported on macOS",
"windows-x64": "CUDA packages not supported on Windows"
}
}
}
```

### Forcing Source Build for Packages Without Binary Wheels

```json
{
"forceSource": {
"parasail": {
"linux-aarch64": "parasail has no binary wheels for Linux ARM64",
"macosx-aarch64": "parasail has no binary wheels for macOS ARM64"
}
}
}
```

### Advanced Platform-Specific Configuration

```json
{
"platformSpecific": {
"tensorflow": {
"linux-aarch64": {
"action": "skip",
"reason": "TensorFlow has limited ARM64 support"
},
"macosx-aarch64": {
"action": "forceSource",
"reason": "TensorFlow ARM64 builds are experimental"
}
}
}
}
```

## Adding New Exceptions

1. Edit `package-exceptions.json`
2. Add your package and platform-specific rules
3. Test the build to ensure the exceptions work as expected
4. Commit the changes

## Best Practices

- **Be specific**: Only add exceptions when absolutely necessary
- **Document reasons**: Always provide clear explanations for why exceptions exist
- **Test thoroughly**: Verify that exceptions work on all affected platforms
- **Keep updated**: Remove exceptions when packages add support for new platforms
- **Use simple configs**: Prefer `skip` and `forceSource` over `platformSpecific` when possible

## Troubleshooting

If the build fails to load the exceptions configuration:

1. Check that `package-exceptions.json` is valid JSON
2. Verify the file is in the project root
3. Check file permissions
4. Look for console warnings during build startup

The build will continue with an empty configuration if the file cannot be loaded.
23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,29 @@ As we do not maintain/build our own java, the version of this package is not bou
specific version of python, but newer python version publications produce newer entrypoints
in this package.

## Package Management

The build system automatically handles package installation with intelligent fallback:
- First attempts to download binary wheels from PyPI
- Falls back to building from source if binary wheels are unavailable
- Supports platform-specific exceptions for packages with limited platform support
- Supports additional PyPI index URLs for custom package sources

### Usage

```bash
# Basic build
npm run build

# Build with additional PyPI index URLs
node ./scripts/build.js 3.12.6 "https://custom.pypi.org/simple/,https://another.pypi.org/simple/"

# Build with single additional index
node ./scripts/build.js 3.12.6 "https://custom.pypi.org/simple/"
```

See [PACKAGE_EXCEPTIONS.md](PACKAGE_EXCEPTIONS.md) for details on configuring package-specific behavior.

## How to release new version of python run environment

1. Update `package.json`:
Expand Down
35 changes: 35 additions & 0 deletions package-exceptions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"skip": {
"cudf-cu12": {
"macosx-x64": "CUDA packages not supported on macOS",
"macosx-aarch64": "CUDA packages not supported on macOS",
"windows-x64": "CUDA packages not supported on Windows"
},
"cupy-cuda12x": {
"macosx-x64": "CUDA packages not supported on macOS",
"macosx-aarch64": "CUDA packages not supported on macOS",
"windows-x64": "CUDA packages not supported on Windows"
}
},
"forceSource": {
"parasail": {
"linux-aarch64": "parasail has no binary wheels for Linux ARM64",
"macosx-aarch64": "parasail has no binary wheels for macOS ARM64"
},
"pynacl": {
"windows-x64": "pynacl often fails to build on Windows, prefer source"
}
},
"platformSpecific": {
"tensorflow": {
"linux-aarch64": {
"action": "skip",
"reason": "TensorFlow has limited ARM64 support"
},
"macosx-aarch64": {
"action": "forceSource",
"reason": "TensorFlow ARM64 builds are experimental"
}
}
}
}
2 changes: 1 addition & 1 deletion packages.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ parasail==1.3.4
numpy==2.2.6
umap-learn==0.5.7
PyYAML==6.0.2
cudf-cu12==25.4.0
cudf-cu12==25.6.0
Loading
Loading