generator

This article needs additional citations for verification. Please help improve this article by adding reliable references. Unsourced material may be challenged and removed. (July 2007)

In computer science, a generator is a special routine that can be used to control the iteration behaviour of a loop. A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator.

Generators can be implemented in terms of more expressive control flow constructs, such as coroutines or first-class continuations.^[1]

History

Generators first appeared in CLU (1975),^[2] were a prominent feature in the string manipulation language Icon (1977) and are now available in Python,^[3] C#, JavaScript,^[4] Ruby and other languages. In CLU and C#, generators are called iterators, and in Ruby, enumerators.

Uses

Generators are usually invoked inside loops.^[5] The first time that a generator invocation is reached in a loop, an iterator object is created that encapsulates the state of the generator routine at its beginning, with arguments bound to the corresponding parameters. The generator's body is then executed in the context of that iterator until a special yield action is encountered; at that time, the value provided with the yield action is used as the value of the invocation expression. The next time the same generator invocation is reached in a subsequent iteration, the execution of the generator's body is resumed after the yield action, until yet another yield action is encountered. In addition to the yield action, execution of the generator body can also be terminated by a finish action, at which time the innermost loop enclosing the generator invocation is terminated.

Because generators compute their yielded values only on demand, they are useful for representing sequences that are expensive to compute, or even infinite.

In the presence of generators, loop constructs of a language can be reduced into a single loop ... end loop construct; all the usual loop constructs can then be comfortably simulated by using suitable generators in the right way.

Python

An example Python generator:

def countfrom(n):
    while True:
        yield n
        n += 1
 
# Example use: printing out the integers from 10 to 20.
# Note that this iteration terminates normally, despite countfrom() being
# written as an infinite loop.
 
for i in countfrom(10):
    if i <= 20:
        print(i)
    else:
        break
 
# Another generator, which produces prime numbers indefinitely as needed.
 
def primes():
    n = 2
    p = []
    while True:
        if not any( n % f == 0 for f in p ): #This works in Python 2.5+ or with numpy's any()
            yield n
            p.append( n )
        n += 1

In Python, a generator can be thought of as an iterator that contains a frozen stack frame. Whenever the iterator's next() method is called, Python resumes the frozen frame, which executes normally until the next yield statement is reached. The generator's frame is then frozen again, and the yielded value is returned to the caller.

Generator Expressions

Python has a syntax modelled on that of list comprehensions, called a generator expression that aids in the creation of generators. The following extends the example above by using a generator expression to compute squares from the countfrom generator function:

squares = ( n*n  for n in countfrom(0) )
 
for j in squares:
    if j <= 20:
        print(j)
    else:
        break

C#

An example C# 2.0 generator:

// Method that takes an iterable input (possibly an array)
// and returns all even numbers.
public static IEnumerable<int> GetEven(IEnumerable<int> numbers)
{
    foreach (int i in numbers)
    {
        if ((i % 2) == 0)
        {
            yield return i;
        }
    }
}

You may even use multiple yield return statements and the compiler will return them in order on each iteration:

public class CityCollection : IEnumerable<string>
{
   public IEnumerator<string> GetEnumerator()
   {
      yield return "New York";
      yield return "Paris";
      yield return "London";
   }
}

Both of these examples utilise Generics, but this is not required. To use the yield keyword, you must be using at least C# version 2.0.

XL

In XL, iterators are the basis of 'for' loops:

import IO = XL.UI.CONSOLE
 
iterator IntegerIterator (var out Counter : integer; Low, High : integer) written Counter in Low..High is
    Counter := Low
    while Counter <= High loop
        yield
        Counter += 1
 
// Note that I needs not be declared, because declared 'var out' in the iterator
// An implicit declaration of I as an integer is therefore made here
for I in 1..5 loop
   IO.WriteLn "I=", I

Tcl

In Tcl 8.6, the generator mechanism is founded on named coroutines.

proc generator {body} {
    coroutine gen[incr ::disambiguator] apply {{script} {
        # Produce the result of [generator], the name of the generator
        yield [info coroutine]
        # Do the generation
        eval $script
        # Finish the loop of the caller using a 'break' exception
        return -code break
    }} $body
}
 
# Use a simple 'for' loop to do the actual generation
set count [generator {
    for {set i 10} {$i <= 20} {incr i} {
        yield $i
    }
}]
 
# Pull values from the generator until it is exhausted
while 1 {
    puts [$count]
}

Ruby

Ruby supports generators (starting from version 1.9) in form of built-in Enumerator class.

# Generator from an Enumerable object
chars = Enumerator.new(['A', 'B', 'C', 'Z'])
 
4.times { puts char.next }
 
# Generator from a block
count = Enumerator.new do|yielder|
  i=0
  loop{ yielder.yield i += 1}
end
 
100.times { puts count.next }

Other Implementations

Java does not have continuation or functionality out of the box, but generators can be implemented using threading. An abstract class that can be subclassed to write generators in Java can be found here.

Another implementation, which uses bytecode enhancements to declared generators instead of threading (specifically, WebObject's ASM) and the new Java 5 feature of VM Instrumentation) was also made public here.

Usage example (for either implementation):

  Generator fibonacci = new Generator() {
      @Override
      public void run() {
            int a = 0, b = 1;
            while (true) {
                a = b + (b = a);
                yield(a);
          }
      }
  };
 
  for (int x : fibonacci) {
      if (x > 20000) break;
      out.println(x);
  }

See also

• List comprehension for another construct that generates a sequence of values
• Iterator for the concept of producing a list one element at a time
• Lazy evaluation for producing values when needed
• Corecursion for potentially infinite data by recursion instead of yield
• Coroutine for even more generalization from subroutine
• Continuation for generalization of control flow

Notes

1. ^ Kiselyov, Oleg (January 2004). "General ways to traverse collections in Scheme". http://okmij.org/ftp/Scheme/enumerators-callcc.html. 
2. ^ Liskov, Barbara (April 1992). "A History of CLU" (pdf). http://www.lcs.mit.edu/publications/pubs/pdf/MIT-LCS-TR-561.pdf. 
3. ^ Python Enhancement Proposals: PEP 255: Simple Generators, PEP 289: Generator Expressions, PEP 342: Coroutines via Enhanced Generators
4. ^ "New In JavaScript 1.7". http://developer.mozilla.org/en/docs/New_in_JavaScript_1.7#Generators. Retrieved 2006-10-10. 
5. ^ The Icon Programming Language utilizes generators to implement its goal directed evaluation. In Icon, generators can be invoked in contexts outside of the normal looping control structures.

References

• Stephan Murer, Stephen Omohundro, David Stoutamire and Clemens Szyperski: Iteration abstraction in Sather. ACM Transactions on Programming Languages and Systems, 18(1):1-15 (1996) [1]