Move UUIDText, dsc, and TimeSync to HashMaps #66
+1,102
−909
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR continues the work done by jrouaix and dgmcdona to reduce this library’s memory usage.
Prior to this PR the library would read and cache all UUIDText and dsc files on a system or logarchive prior to parsing the actual logs.
The UUIDText and dsc files contains part of the log message.
This PR changes that behavior.
Instead, the library will parse the log file first and then read the corresponding UUIDText and/or dsc file.
The library will then cache the the corresponding UUIDText and/or dsc file before moving on to the next log entry.
The library will keep a small cache of UUIDText and dsc files while reading the log entries. This is all handle via the
FileProvider
trait that dgmcdona addedCurrently it will cache 30 UUIDText files and 2 dsc files.
There typically ~250 UUIDText files. With a total size of ~90MB to ~140MB
Dsc files are larger but fewer in number.
Typically ~6 dsc files. With a total size of ~600MB
By using a smaller cache, the memory usage of the library drops a pretty good amount.
Memory usage prior to this PR: ~300MB-1GB. Dependent on how many UUIDText and dsc files
Now: ~100MB-500MB.
(Though this is still higher than the builtin in
log
command Apple. Which is ~150MB - ~200MB)Switching to HashMap also makes the library a bit faster now
basic hyperfine test on a few log files
Where
iteratorV2
is the HashMap versionFinally this PR includes a fix to better handle
<private>
number entries.Also since we switch from Vec to HashMap this is probably a breaking change. The example file has been updated.