Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash with --replace-text input file longer than ~1550 lines (StackOverFlowError) #528

Open
jasalt opened this issue Jan 23, 2025 · 1 comment
Open

Crash with --replace-text input file longer than ~1550 lines (StackOverFlowError) #528

jasalt opened this issue Jan 23, 2025 · 1 comment

Comments

@jasalt
Copy link

jasalt commented Jan 23, 2025

Seems that program crashes with input file longer than 1563 lines when using --replace-text option.

System info:

java --version
openjdk 17.0.13 2024-10-15
OpenJDK Runtime Environment (build 17.0.13+11-Debian-2deb12u1)
OpenJDK 64-Bit Server VM (build 17.0.13+11-Debian-2deb12u1, mixed mode, sharing)
java -jar ./bfg-1.15.0.jar --version
bfg 1.15.0

Reproduction:

git clone https://github.com/rtyley/bfg-repo-cleaner.git
cd bfg-repo-cleaner
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.15.0/bfg-1.15.0.jar

# make random numbers in file
< /dev/urandom tr -dc '0-9' | fold -w 13 | head -n 10000 > 10000_random_numbers.txt


java -jar ./bfg-1.15.0.jar --replace-text 10000_random_numbers.txt .

Crashes with:

$ java -jar ./bfg-1.15.0.jar --replace-text 10000_random_numbers.txt .

Using repo : /home/user/tmp/bfg-repo-cleaner/./.git


This repo has been processed by The BFG before! Will prune repo before proceeding - to avoid unnecessary cleaning work on unused objects...
Completed prune of old objects - will now proceed with the main job!

Found 122 objects to protect
Found 57 commit-pointing refs : HEAD, refs/heads/main, refs/remotes/origin/HEAD, ...
Found 58 tag-pointing refs : refs/tags/v0.10.0, refs/tags/v0.11.0, refs/tags/v0.12.0, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at use.newer.java.Util.invokeMainMethod(Util.java:89)
        at use.newer.java.Util.executeMainProgram(Util.java:28)
        at use.newer.java.Util.checkJavaVersionAndExecuteMainProgram(Util.java:18)
        at use.newer.java.Version8.main(Version8.java:5)
Caused by: com.google.common.util.concurrent.ExecutionError: com.google.common.util.concurrent.ExecutionError: java.lang.StackOverflowError
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2084)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
        at com.madgag.git.bfg.MemoUtil$$anon$2.apply(memo.scala:60)
        at com.madgag.git.bfg.GitUtil$$anon$1.apply(GitUtil.scala:71)
        at com.madgag.git.bfg.CleaningMapper.replacement(GitUtil.scala:46)
        at com.madgag.git.bfg.CleaningMapper.replacement$(GitUtil.scala:45)
        at com.madgag.git.bfg.GitUtil$$anon$1.replacement(GitUtil.scala:70)
        at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.$anonfun$reportsFor$3(ProtectedObjectDirtReport.scala:45)
        at scala.util.Either.fold(Either.scala:197)
        at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.$anonfun$reportsFor$1(ProtectedObjectDirtReport.scala:45)
        at scala.collection.Iterator$$anon$9.next(Iterator.scala:584)
        at scala.collection.mutable.Growable.addAll(Growable.scala:62)
        at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
        at scala.collection.immutable.SetBuilderImpl.addAll(Set.scala:405)
        at scala.collection.immutable.Set$.from(Set.scala:362)
        at scala.collection.immutable.Set$.from(Set.scala:93)
        at scala.collection.IterableOps.map(Iterable.scala:684)
        at scala.collection.IterableOps.map$(Iterable.scala:684)
        at scala.collection.AbstractIterable.map(Iterable.scala:935)
        at com.madgag.git.bfg.cleaner.protection.ProtectedObjectDirtReport$.reportsFor(ProtectedObjectDirtReport.scala:43)
        at com.madgag.git.bfg.cleaner.CLIReporter.reportProtectedCommitsAndTheirDirt(Reporter.scala:133)
        at com.madgag.git.bfg.cleaner.CLIReporter.reportObjectProtection(Reporter.scala:106)
        at com.madgag.git.bfg.cleaner.RepoRewriter$.rewrite(RepoRewriter.scala:96)
        at com.madgag.git.bfg.cli.Main$.$anonfun$new$1(Main.scala:59)
        at com.madgag.git.bfg.cli.Main$.$anonfun$new$1$adapted(Main.scala:34)
        at scala.Option.map(Option.scala:242)
        at com.madgag.git.bfg.cli.Main$.delayedEndpoint$com$madgag$git$bfg$cli$Main$1(Main.scala:34)
        at com.madgag.git.bfg.cli.Main$delayedInit$body.apply(Main.scala:27)
        at scala.Function0.apply$mcV$sp(Function0.scala:42)
        at scala.Function0.apply$mcV$sp$(Function0.scala:42)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:98)
        at scala.App.$anonfun$main$1$adapted(App.scala:98)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
        at scala.App.main(App.scala:98)
        at scala.App.main$(App.scala:96)
        at com.madgag.git.bfg.cli.Main$.main(Main.scala:27)
        at com.madgag.git.bfg.cli.Main.main(Main.scala)
        ... 8 more
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.StackOverflowError
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2084)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4017)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4040)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4989)
        at com.madgag.git.bfg.MemoUtil$$anon$2.apply(memo.scala:60)
        at scala.collection.immutable.List.map(List.scala:247)
        at scala.collection.immutable.List.map(List.scala:79)
        at com.madgag.git.bfg.cleaner.TreeBlobModifier.apply(TreeBlobModifier.scala:37)
        at com.madgag.git.bfg.cleaner.TreeBlobModifier.apply$(TreeBlobModifier.scala:37)
        at com.madgag.git.bfg.cli.CLIConfig$$anon$2.apply(CLIConfig.scala:174)
        at com.madgag.git.bfg.cli.CLIConfig$$anon$2.apply(CLIConfig.scala:174)
        at scala.Function$.$anonfun$chain$2(Function.scala:23)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at scala.Function$.$anonfun$chain$1(Function.scala:23)
        at com.madgag.git.bfg.cleaner.ObjectIdCleaner.$anonfun$cleanTree$1(ObjectIdCleaner.scala:125)
        at com.madgag.git.bfg.MemoUtil$$anon$3.load(memo.scala:74)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3574)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2190)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2080)
        ... 49 more
Caused by: java.lang.StackOverflowError
        at scala.Function$.$anonfun$chain$2(Function.scala:23)
        at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
        at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
        at scala.collection.immutable.List.foldLeft(List.scala:79)
        at scala.Function$.$anonfun$chain$1(Function.scala:23)
        at scala.Function$.$anonfun$chain$2(Function.scala:23)

End of stack trace repeats itself, full version: https://pastebin.com/AKYcSNU2

As workaround, splitting to parts of 1000 works:

split -l 1000 10000_random_numbers.txt chunk_of_1000_ --additional-suffix=.txt

for file in chunk_of_1000_*.txt; do
    java -jar ./bfg-1.15.0.jar --replace-text "$file" .
done

This evaluates without crash, looping over each chunk of 1000 in bash. Traced down that the error starts happening at around 1564 lines or more while 1563 is still okay.

Not familiar with Scala but could it be recursion limit issue or something?
@jasalt jasalt changed the title Crash with --replace-text input file longer than 1563 lines (StackOverFlowError) Crash with --replace-text input file longer than ~1550 lines (StackOverFlowError) Jan 23, 2025
@jasalt
Copy link
Author

jasalt commented Jan 23, 2025 •
edited
Loading

Exact line number where crash starts happening seems to change somewhat, example input file crashing with 1551 lines https://pastebin.com/raw/UbU4MYHX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jasalt