v0.0.75

Release 0.0.75

What's New

1. Performance Optimization

• Implemented advanced algorithms to enhance processing speed
• Reduced memory footprint for more efficient resource utilization
• Optimized critical paths to improve overall system responsiveness

2. Enhanced NF4 Data Type

• Improved precision and accuracy of NF4 operations
• Optimized NF4 conversions for better performance

3. Structural Enhancements

• Refactored core components for improved modularity
• Introduced a more intuitive project structure for easier navigation and maintenance
• Updated dependency management for better compatibility and future-proofing

v0.0.70

FJFormer 0.0.70 Release Notes

Major Improvements

Documentation

• Comprehensive overhaul of documentation for better clarity and usability
• Added more examples and use cases to help users get started quickly
• Improved API references for easier navigation

LoRA (Low-Rank Adaptation)

• Fixed critical issues affecting LoRA functionality
• Introduced a new option to selectively save only A, B, and Alpha matrices in LoRA instead of the entire merged model
  • This feature significantly reduces storage requirements and improves flexibility in model management

Checkpoint Managers

• Updated with enhanced functionality for more efficient model state handling
• Improved integration with distributed training workflows

Removed Features

• The FJFormer Linen API has been deprecated and removed from this version

New Features

ImplicitArray

• Using core.ImplicitArray, a powerful abstraction for handling large arrays without instantiation
• Enables lazy evaluation and efficient array operations in JAX

Quantization Support

• Added Array8Bit for 8-bit quantization
  • Reduces model size while maintaining good performance
• Introduced Array4Bit for 4-bit quantization (NF4)
  • Offers extreme model compression for scenarios where size is critical

0.0.40

Summery

1. Fixing Splash Attention Kernel
2. Improving project structure
3. adding new loss function useable and related to MoE Models

What's Changed

• Add compute_weighted_cross_entropy_and_accuracy() by @yhavinga in #1
• Fix Cosine with Warmup Scheduler by @w11wo in #2

New Contributors

• @yhavinga made their first contribution in #1
• @w11wo made their first contribution in #2

Full Changelog: v0.0.22...v0.0.40

v0.0.22

Full Changelog: v0.0.16...v0.0.22

v0.0.16

Full Changelog: https://github.com/erfanzar/FJFormer/commits/v0.0.16