forked from NotAdam/Lumina
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suggestions on Excel #1
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* `IExcelRow.RowId` and `IExcelRow.SubrowId` exist to implement `ICollection<T>.Contains`. * `System.Collections.Generic.EnumerableHelpers.ToArray` and alike in LINQ has an optimization for `ICollection<T>`. As it requires `Contains` to be implemented, exposing `RowId` and `SubrowId` in a generic way will make it possible to implement that in O(1). * `ExcelSheet` constructor now preallocates lookup lookup tables. * `.exh` file comes with information on how many rows are there, so we know the exact number of items that needs to be allocated. * Using an array directly bypassing list wrappers may provide an additional speed boost. * In case `.exh` file contains a wrong information on number of rows, which is an unlikely case, `Array.Resize` is used to reallocate the array. * `ExcelSheet.UnsafeCreateRow/Subrow/At` has been added. * These functions assume that boundary checks are done by callers. * As enumerators always work inside the boundary, especially when the collection is immutable, `IEnumerator{T}.Current` can skip boundary checks. * `DefaultExcelSheet<T>` and `SubrowsExcelSheet<T>` has been added. * As sheets are usually not meaningful without knowing what is in it in the first place, it would be better to specialize for each variants. * This effectively hides subrow operations from sheets of default variants. * This removes `HasSubrows` check from getter functions. * Added `SubrowsExcelSheet.Try/GetRow/OrDefault` variants that returns `SubrowCollection<T>` instead. * This makes it convenient to iterate over subrows under one row ID. * This makes it faster to access multiple subrows under one row ID, as lookup operation is done on obtaining the collection. Once the collection is constructed, accessing subrows is an O(1) operation. * `ExcelModule.GetSheet` uses static lambda in place of `SheetCache.GetOrAdd`. * This will avoid heap allocation if a corresponding sheet is already loaded. * Named value tuples in `ExcelSheet` are replaced with `record struct`. * This reduces the size of each lookup element from 16 bytes to 12 bytes.
Most of sheets do not have large gaps across items. That fact can be used to make a lookup array instead of lookup dictionary, which will even further reduce the time spent translating row ID to row index.
Added lookup array to use instead of dictionary, if the wasted memory for lookup array items that resolve to nonexistence are not too huge. Code used to evaluate wasted memory usage, to be run as a test. [RequiresGameInstallationFact]
public void TestAllSheets()
{
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Highest;
var gameData = new GameData( @"C:\Program Files (x86)\SquareEnix\FINAL FANTASY XIV - A Realm Reborn\game\sqpack" );
var gaps = new List< uint >();
foreach( var sheetName in gameData.Excel.SheetNames )
{
if( gameData.GetFile< ExcelHeaderFile >( $"exd/{sheetName}.exh" ) is not { } headerFile )
continue;
var lang = headerFile.Languages.Contains( Language.English ) ? Language.English : Language.None;
switch( headerFile.Header.Variant )
{
case ExcelVariant.Default:
{
var sheet = (DefaultExcelSheet<Addon>) ExcelSheet.From< Addon >( gameData.Excel, lang, sheetName );
gaps.Add( sheet.Count == 0 ? 0 : sheet.GetRowAt( sheet.Count - 1 ).RowId - sheet.GetRowAt( 0 ).RowId + 1 - (uint)sheet.Count );
break;
}
case ExcelVariant.Subrows:
{
var sheet = (SubrowsExcelSheet<QuestLinkMarker>) ExcelSheet.From< QuestLinkMarker >( gameData.Excel, lang, sheetName );
gaps.Add( sheet.Count == 0 ? 0 : sheet.GetRowAt( sheet.Count - 1 ).RowId - sheet.GetRowAt( 0 ).RowId + 1 - (uint)sheet.Count );
break;
}
}
}
gaps.Sort();
var countAcc = 0;
var wasteAcc = 0;
var test = string.Join(
"\n",
gaps
.GroupBy( static x => x / 1024, static x => x )
.Select( x =>
$"{( x.Key + 1 ) * 1024,8}, {( countAcc += x.Count() ) * 100f / gaps.Count,6:00.00}%, {( wasteAcc += x.Sum( static y => (int) y ) * 4 ) / 1024,5}KB" ) );
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IExcelRow.RowId
andIExcelRow.SubrowId
exist to implementICollection<T>.Contains
.System.Collections.Generic.EnumerableHelpers.ToArray
and alike in LINQ has an optimization forICollection<T>
. As it requiresContains
to be implemented, exposingRowId
andSubrowId
in a generic way will make it possible to implement that in O(1).ExcelSheet
constructor now preallocates lookup lookup tables..exh
file comes with information on how many rows are there, so we know the exact number of items that needs to be allocated..exh
file contains a wrong information on number of rows, which is an unlikely case,Array.Resize
is used to reallocate the array.ExcelSheet.UnsafeCreateRow/Subrow/At
has been added.IEnumerator{T}.Current
can skip boundary checks.DefaultExcelSheet<T>
andSubrowsExcelSheet<T>
has been added.HasSubrows
check from getter functions.SubrowsExcelSheet.Try/GetRow/OrDefault
variants that returnsSubrowCollection<T>
instead.ExcelModule.GetSheet
uses static lambda in place ofSheetCache.GetOrAdd
.ExcelSheet
are replaced withrecord struct
.Benchmark
Test Code