Description
Problem 6. (Automatic Writing)
Writing-intensive modules can be hard: so many 10-page essays, and not nearly enough time to
catch up on the latest e-lectures (please watch them!). CS2040S is here to help. For this problem
set, you will develop an automatic writing program that can easily produce pages and pages of new
text. And it will adapt to your chosen style. If you use an old essay as input, your new essay will
sound just like it was written by you! If you use Shakespeare as input, your new essay will sound
as if it was written by the Bard himself.
The basic idea is to take an input text and calculate its statistical properties. For example, given
a specific string “prope”, what is the probability that the next letter is an ‘r’? What is the
probability that the next letter is a ‘q’? Your program will take a text as input, calculate this
statistical information, and then use it to produce a reasonable output text.
Claude Shannon first suggested this technique in a seminal paper A Mathematical Theory of Communication (1948). This paper contained many revolutionary ideas, but one of them was to use
a Markov Chain to model the statistical properties of a particular text. Markov Chains are now
used everywhere; for example, the Google PageRank algorithm is built on ideas related to Markov
Chains.
Markov Models
Given a text, you can build up the Markov Model. A Markov Model captures the frequency of
a specific letter/character appearing after a specific preceding string (which can be of varying
length). The order of the Markov model is the length of that preceding string.
For example, if we have the following text:
a b d a c a b d a c b d a b d a c d a
We can build the following Markov Model of order 1:
a b 1/2
a c 1/2
b d 1
c a 1/3
c b 1/3
c d 1/3
d a 1
This implies the following:
• After the string ‘a’, half the time you find a ‘b’, and half the time you find a ‘c’.
2
• After the string ‘b’, you always find a ‘d’.
• After the string ‘c’, one-third of the time you find letters ‘a’, ‘b’, or ‘d’ (i.e., they are equally
common after a ’c’).
• After the string ‘d’, you always find an ‘a’.
You can think of these as probabilities (though so far, there is no randomness at all). Notice that
in the text above the table, there are three instances when the character ‘a’ is followed by a ‘b’, and
there are three instances when ‘a’ is followed by a ‘c’. Similarly, ‘b’ is always followed by a ‘d’, and
‘d’ is always followed by an ‘a’. The character ‘c’ is followed by an ‘a’ once, a ‘b’ once, and a ‘d’ once.
A Markov Model of order 2 captures how likely a given letter is to follow a string of length 2.
Suppose we have the following text:
a b c d a b d d a b c d d a b d
Here we have an example of Markov Model of order 2 built by the text above:
ab c 1/2
ab d 1/2
bc d 1
bd d 1
cd a 1/2
cd d 1/2
da b 1
dd a 1
Notice that in the text above, there are two instances when the string ’ab’ is followed by the letter
‘c’ and two instances when the string ’ab’ is followed by the letter ‘d’. After the string ‘bc’, you
always get the letter ‘d’, and after the string ‘bd’, you always get the letter ‘d’, etc.
Producing a New Text
Once you have your Markov Model, you can go about generating a new text. You need to start
with a seed string of the same length as the order of the Markov Model. For example, if the Markov
Model is of order 6, you need to start with a string of length 6.
We use the term ’kgrams’ to refer to the k-character strings, where k is the order. In
order to generate the next character, you look back at the previous k characters (inclusive of the
current last character). Look up that kgram in your Markov Model, and find the frequency that
each character appears after that kgram. If the kgram never appeared in your Markov Model, then
your newly-generated text is completed. Otherwise, you randomly choose the next character based
on the probability distribution indicated by the Markov Model.
3
Once you have found the next character that way, you add it to the end of your string, and repeat
the process as many times as you want!
Problem Details
For this problem, you have been provided with the TextGenerator Java class, more information
will be provided in the section below. You will submit one Java class: MarkovModel.
Problem 6.a. Implement only the following 4 methods:
MarkovModel(int order, long seed): Constructs a Markov Model of the specified order.
You can assume that the order will be at least 1. The seed should be used to initialize the
pseudorandom number generator that your class will use (the template code already does
this). A pseudorandom number generator’s number sequence is completely determined by
the seed. So, if a pseudorandom number generator is reinitialized with the same seed, it will
produce the same sequence of numbers.
void initializeText(String text): This builds your Markov Model based on the specified
text. When this routine is complete, your class would have computed the table that maps
each kgram to the frequency that each ASCII character follows that kgram (where k is the
order of the Markov Model).
int getFrequency(String kgram): Returns the number of times the specified string kgram
appears in the input text. The behaviour of this method is only defined if the length of
the kgram is equal to the order of the Markov Model. This should return a number zero or
greater. For example, in the Figure 1 below, the frequency of kgram ’aa’ is 1. (Notice we do
not count the kgram which appears at the end of the text, where nothing may follow it)
int getFrequency(String kgram, char c): Returns the number of times the specified
character c appears immediately after the string kgram in the input text. The behaviour of
this method is only defined if the length of the kgram is equal to the order of the Markov
Model. For example, in the Figure 1 below, the frequency of ’g’ appearing after the kgram
’ga’ is 4. If the string kgram never appears in the original text, then you should return 0.
Problem 6.b. Next, implement the nextCharacter method:
char nextCharacter(String kgram): Returns a random character chosen according to the
Markov Model. The probability of a character ‘c’ should be equal to getFrequency(kgram,c)
getFrequency(kgram) .
That is, the probability of character ‘c’ should be equal to the frequency that ‘c’ follows the
string kgram in the text. If there is no possible next character (e.g., because the kgram does
not appear in the text, or only appears at the very end of the text), then return the specially
4
defined token final char NOCHARACTER = 0 (defined in the template file). The kgram must
be the length specified by the order of the Markov model. To generate the random choice,
you must use the pseudorandom number generator with the specified seed. You must use
the process described in ”Random character generation” below to generate the
random choice.
You may assume that every character in the text is an ASCII character (inclusive of the extended
ASCII character), except for 0 (the NULL character), which does not count as a character. Figure 1
has an example of the information computed by your Markov Model for a specific example string.
frequency probability
a c g a c g
aa 1 1 0 0 1 0 0
ag 4 2 0 2 2/4 0 2/4
cg 1 1 0 0 1 0 0
ga 5 1 0 4 1/5 0 4/5
gc 1 0 0 1 0 0 1
gg 3 1 1 1 1/3 1/3 1/3
Figure 1: Markov Model produced by the string gagggagaggcgagaaa.
Implementation advice
There are several approaches to designing the MarkovModel class. Here we provide some tips for
how you might implement it.
Basic structure.
There are two standard ways you might store the information about the kgram. You might use a
symbol table (i.e., a hash table) that maps strings of length k (where k is the order of the Markov
model) to an array containing 255 integers, one representing each possible ASCII character. The
array records the number of time each character follows the given string. For example, the character
‘a’ is 97, in ASCII. Hence, if k = 2, given an input string ‘xya’, you would add to your hash table an
entry with the key equal to ‘xy’ and the value equal to an array of integers where value[97] == 1.
Alternatively, you might use a symbol table that maps strings of length k to another symbol table;
the second symbol table maps ASCII characters to integers. Again, these integers represent the
frequency that a given character follows the initial string. You may also use an alternate solution,
as long as it efficiently supports the required operations.
For the purposes of this Problem Set, you should use a Hash Table and NOT use Tries or
TreeSet or TreeMap to store information about the kgram. If you are unsure whether you can
use a particular data structure, please post in the forums to seek clarifications.
5
Random character generation
Presumably you have now built your Markov Model, and now know the proper frequencies that
each character is followed by each kgram. In order to generate the next character in the text, you
need to make a random choice. For testing purposes, you must use the random number generator
specified in the template code: java.util.Random, and you must use the seed specified in the
constructor (via a setSeed call on the random number generator object which is already in the
template code). If you run the same test twice with the same seed, you will get the same answer!
This is very useful for testing.
A second requirement (for testing purposes) is that you choose the next character randomly in the
following way. Here is an example. Imagine that for the given kgram, there are four characters that
might follow it: ‘j’, ‘m’, ‘p’, ‘z’ (all the other characters appear zero times after this kgram). The
‘j’ appears 2 times, the ‘m’ appears 5 times, the ‘p’ appears 3 times, and the ‘z’ appears 1 time.
Thus this particular kgram appears 11 times (not counting the end of the text, where nothing may
follow it).
To choose a random character, you first select a random integer from [0, 10], i.e., a range containing
11 integers. You can do this by calling generator.nextInt(11). You can then partition this range
of random numbers:
• if you get {0, 1} then you return ’j’;
• if you get {2, 3, 4, 5, 6}, then you return ’m’;
• if you get {7, 8, 9}, then you return ‘p’;
• if you get {10}, then you return ‘z’.
You must process the possible letters in alphabetic order (or by order of their ASCII character
values).
In general, if there is a set of C possible next characters which appear a total of N times after the
k-letter prefix, you should choose a random number from [0, N − 1], and then go through the set
in alphabetical order to determine which character was chosen by the random selection, weighting
each character by the number of times it appears.
The reason we ask you to choose the random character in this way is twofold: first, it is a reasonably
efficient way to sample from a distribution, and second, it will allow us to ensure that every solution
produces the same random sequence (which makes it easier to test).
Other tips
Here are a few other things that may be useful:
• You can treat an ASCII character either as a char or an int. If you have a char ch, you can
store it in an integer as follows: int iChar = (int) ch. Similarly, you can do the opposite:
6
char newChar = (char) iChar. In general, it is not recommended that you force Java to
ignore types like this. However, in this case, as long as you are careful to not use any values
about 255, this is a safe thing to do. (If your integers is bigger than 255 and you put it in
a char, you may not get what you expect.) This is convenient, for example, if you want to
enumerate all the ASCII characters using a for loop, since you can just count up to 255.
• A Java HashMap is a parameterized data type. That means that when you create it, you
have to specify what the key and value types are. For example, if you want to use a key type
K and a value type V, you would use the class HashMap
want a key type of String and a value type of Integer, you use a HashMap
(Whenever you need to use the name of the class, whether to declare the variable or to create
a new object, you can just use that full name including the String and Integer types.)
• Notice, though, that for your key and value types, you cannot use primitive types like int or
char. Instead, you have to use the wrapper class version of these types: Integer, Character,
etc. That is why, above, we used Integer as the value type. You can mostly use int and
Integer interchangeably and everything just works, with Java automatically converting back
and forth between the two as needed. (An Integer is just a class that contains inside it an
int.)
• Once you have declared your hashmap with the key and value types as String and Integer,
then when you use the put and get methods, they act just like they should, e.g., get takes
a String as input and returns an Integer as output.
• You might find a variety of the methods in the String class useful. If you have not already,
look them up in the Java documentation. For example, you can use the charAt method to
retrieve a particular character in a string; you can use substring(int j) to retrieve the
suffix of a string (from j onwards), and you can use substring(int i, int j) to retrieve
the substring between character i (inclusive) and j (exclusive). Do look up the details.
Text Generator
We have provided you with a text generator class that you can use with your Markov Model class
to generate text. The text generator class takes three input parameters, i.e., the main method has
argument (arg[0], arg[1], arg[2]):
• k, the order of the Markov model;
• n, the number of characters to generate;
• the filename of the text to use as a model. (This file should be in the project root directory,
above the src folder.)
You can set the input parameters in IntelliJ under Run → Edit Configurations, where you can
create a new configuration (of type Application). There you can set the program arguments. For
example: ”3 15 PS6Test.in” as the program arguments.
7
The text generator reads in the file as a long text String, creates the MarkovModel, and calls
initializeString on your Markov Model class using the text.
It then generates a new text, using the first k characters of the original input text as a seed (and
as the first k characters of the output text). In more detail, it begins with a String kgram equal
to the first k letters from your text file. It generates the next character by calling nextCharacter
on your Markov Model, using the initial kgram as your input string. Then it updates the kgram,
adding the new character to the end. It continues in this way, at every step using the most recent
k characters to query the Markov Model for the next character.
If it ever reaches a point where there is no possible next character, then it stops (outputting a
string shorter than desired).
We will test the functionality of your nextCharacter method in Problem 6.b. by making use of
this TextGenerator class that we have provided for you.
Optional Experiments
You may want to experiment with different values of k to determine which values yield reasonable
sounding texts. If k is too large, then the text output will be identical to the input text. On the
other hand, if k is too small, then the output text is quite garbled and ungrammatical English.
There is a small range of k for which the output text both sounds like real English, but also sounds
like a new and unique text.
Problem 6.c. (BONUS) Word-based Markov Model:
You might also experiment with using words instead of characters. For example, instead of looking
at the probability that character ‘c’ comes after the string ‘cra’, you could look at the probability
that the word cat comes after the word yellow (i.e., order 1, where order is defined based on the
number of words), or the probability that cat comes after the phrase the vicious yellow (i.e., order
3). If you develop your Markov Model based on words, you might get a more interesting text, as
long as you begin with a sufficiently long text.
In order to use words instead of characters, you will have to design your MarkovModel class more
carefully. You are now not restricted to the same scheme as the Random character generation, and
you may wish to explore other schemes for random word generation.
Submit your well-commented MarkovModel class to Coursemology. You are also to upload your
(potentially) modified TextGenerator class, OR any other class which makes use of your modified
MarkovModel class to generate text. This is to allow your tutors to test your implementation for
this problem.
Note: This bonus problem is an extension to the basic character-based Markov Model. You must
still submit your basic version, and it must work reasonably well before your submission for
this word-based Markov Model is considered for bonus marks.
8
Creative Competition
Submit the best, most interesting text that your program produces. Post your best, most creative
work to the forum (in the specified location). The one(s) with the most upvotes win(s)! You could
aim for several goals: (i) plausibility (i.e., does it read like a real text), (ii) novelty, and (iii) humor.
In order to generate an interesting text, you will need to find a good (long) source text. In the
past, people have used legal documents (e.g., the text of a specific law), Shakespeare, news reports,
and/or arbitrary texts from Project Gutenberg (gutenberg.org). You may splice together 2-3
input texts. You may also experiment with modifying the initial kgram. Instead of using the first k
letters of the text as an initial kgram, you may choose an alternate choice of k characters. (Specify
whether your entry was generated using characters, as in the assignment, or using words, as in the
extension, or using something else.)
On a closing note:
“Half the fire a funny little man began to stay at heavens. ‘How beautiful this kingdom’
said their new emperor.”
9
