forEach java 8 throws StackOverflow Error for big dataset

Question

I am trying soomething like -

        Stream<String> wat = Stream.empty();
        for (long i=0;i<10000000L;i++) {
            Stream<String> yaya = Stream.of("This iis a very biig string lsjdflkjkj lkasdjf lkjdsal");
            wat = Stream.of(wat, yaya).flatMap(Function.identity());
        }

        AtomicInteger i= new AtomicInteger();

        wat.forEach(st-> {
            i.incrementAndGet();
        });

        System.out.println(i);

The above throws stack overflow error -

java.lang.StackOverflowError: null
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485) ~[?:1.8.0_275]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272) ~[?:1.8.0_275]
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_275]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_275]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_275]
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_275]

After few hit and trial i was able to resolve this problem by using list of streams and using nested for loop -

            List<Stream<String>> listOfStream = new ArrayList<>();
            for (long i=0;i<10000000L;i++) {
                Stream<String> yaya = Stream.of("This iis a very biig string lsjdflkjkj lkasdjf lkjdsal");                
                listOfStream.add(yaya);
            }

            AtomicInteger i= new AtomicInteger();
            listOfStream.forEach(st ->
                            st.forEach(sm -> {
                                i.incrementAndGet();
                            })
            );
            System.out.println(i);

This code doesn't throw any stack overflow issue. I've tried to look up various articles but couldn't comprehend why this is working and the former fails. Please help.

Answer 1

What you're doing in the initial for loop is to build a very big object tree describing what you want to achieve.

There will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, one of which will be a Stream implementation that knows that it wants to do a flatMap on two values, ... , one of which is an empty Stream implementation.

Internally those are implemented as wrapper around each other and by calling each other when necessary.

Since each Stream implementation needs to call the next one and that chain is many thousands of objects deep, you'll need at least such a deep stack to be able to fully execute it (in practice probably a multiple of that number, since there might be some helper/intermediary methods on the stack as well).

In your working code you only have one Stream that holds a list of Stream objects to iterate over. So the call depths is very limited: the outer stream delegates to the inner ones one-after-another.