For the 340, I found all period x bigram repeats where x is a divisor of the position spacing of the repeat. Here is the table. On the left is the period. On the right are the position spacings. I must have gotten lucky when checking period 26 before, as I did not find any other period with such a high count of the same position spacings.
There are a lot of period 2 and period 3 bigram repeats where the period is a divisor of the position spacing of the repeats. But that is to be expected, because a lot of numbers can be divided by 2 or 3.
Besides the period 26 bigram repeats, I don’t see anything particularly interesting.
Interesting! I tried to run the same experiment, but instead of using the original position numbers, I used the new position numbers of the bigrams as they repeat in the untransposed ciphertext. I found an unusual spike at period 39, where some of the bigram repeats have positions that share a common factor of 25. I tested periods 1 through 170, and the average number of bigram repeat spacings sharing a factor of 25 was only 0.84, with a standard deviation of 0.99. At period 39, there are 7 bigrams whose position spacing have a common factor of 25. That’s 6.22 standard deviations from the mean!
To visualize this, untranspose Z340 at period 39, then write it to 25 columns like this:
Then you can see these repeating bigrams lining up by columns:
Some of the bigrams overlap. Makes me wonder if that causes overcounting.
For comparison, here are all the repeating bigrams at period 39:
Also, these "triples" stood out to me:
I ran the same experiment on Z408 for all periods and the untransposed cipher (period 1) has a strong spike for factor 17, corresponding to the bigrams that line up along columns. Here’s what they look like in the cipher:
And the plaintext:
Seems to be a mix of naturally occurring bigrams (that seem to line up by chance), and the "filler" section at the end.
Perhaps some of the untransposition schemes for Z340 are uncovering some echoes of the similarly naturally occurring bigrams in the Z340’s plaintext.
I am glad that you are bringing this up, not just railfence but any transposition idea, because there is something that I have wanted to do for a while and haven’t gotten around to it.
Below are the period 19 bigram repeat positions shaded yellow. The message on the left and right are both the 340, but I show it twice for clarity. There are a lot of consecutive diagonal shaded positions. Some are shown boxed blue on the left, and others boxed green on the right for clarity. I made the picture last year, and I think that I only boxed where there are at least three consecutive diagonal ( period 19 ) positions.
If you untranspose the cipher text at period 19, the "consecutive diagonal positions" can be interpreted as repeating bigrams that are directly adjacent (or overlapping) with one another:
If I’m not mistaken, any area there with more than 2 symbols in a row highlighted will show up in the unmodified Z340 as consecutive diagonals. Here’s another view, with the non-consecutive bigrams outlined instead of highlighted:
Those graphics look awesome.
They are great visualization but i find the 408 experiment very interesting. Thanks for sharing.
I am going to rest a little bit tonight. But I think that the next step here should probably be to look at some of the messages in doranchak’s library to see if we can find equally strange spacings of period x bigram repeats. Here are the message numbers:
202
203
222
301
302
318
329
337
338
All have coincidence count spikes of 18 or more.
The below is two row rail fence forward and then coming back then forward back then, two new lines. this brings in the 19 periods.
1 6 47 50 79 72 104 145 150 185 178 207 243 248 289 284 313
20 25 66 65 94 87 123 164 169 200 193 222 262 267 306 299 328
5 10 51 46 75 103 108 149 152 181 174 206 247 252 287 280 309
24 29 68 61 90 122 127 168 167 196 189 225 266 271 302 295 324
9 14 49 42 71 107 112 153 148 177 205 210 251 254 283 276 308
28 33 64 57 86 126 131 170 163 192 224 229 270 269 298 291 327
13 16 45 38 70 111 116 151 144 173 209 214 255 250 279 307 312
32 31 60 53 89 130 135 166 159 188 228 233 272 265 294 326 331
17 12 41 69 74 115 118 147 140 172 213 218 253 246 275 311 316
34 27 56 88 93 134 133 162 155 191 232 237 268 261 290 330 335
15 8 37 73 78 119 114 143 171 176 217 220 249 242 274 315 320
30 23 52 92 97 136 129 158 190 195 236 235 264 257 293 334 339
11 4 36 77 82 117 110 139 175 180 221 216 245 273 278 319 322
26 19 55 96 101 132 125 154 194 199 238 231 260 292 297 338 337
7 35 40 81 84 113 106 138 179 184 219 212 241 277 282 323 318
22 54 59 100 99 128 121 157 198 203 234 227 256 296 301 340 333
3 39 44 85 80 109 137 142 183 186 215 208 240 281 286 321 314
18 58 63 102 95 124 156 161 202 201 230 223 259 300 305 336 329
2 43 48 83 76 105 141 146 187 182 211 239 244 285 288 317 310
21 62 67 98 91 120 160 165 204 197 226 258 263 304 303 332 325
For the 340, I found all period x bigram repeats where x is a divisor of the position spacing of the repeat. Here is the table. On the left is the period. On the right are the position spacings. I must have gotten lucky when checking period 26 before, as I did not find any other period with such a high count of the same position spacings.
There are a lot of period 2 and period 3 bigram repeats where the period is a divisor of the position spacing of the repeats. But that is to be expected, because a lot of numbers can be divided by 2 or 3.
Besides the period 26 bigram repeats, I don’t see anything particularly interesting.
OK I ran another test based on your idea. For all periods (from 1 to 170), I dumped all repeating bigrams and their spacing values (number of positions between repeating bigrams). Then I counted totals of spacing values for all periods. I expected them to be somewhat well-distributed. But the really surprising result is that the spacing value of 78 dominates all the other spacing values. Here are the top 30 results:
78: 161
35: 112
218: 57
71: 52
129: 50
32: 45
226: 43
93: 42
84: 42
162: 42
110: 42
48: 41
75: 39
39: 39
150: 39
210: 35
83: 34
259: 34
137: 34
136: 34
The first number is the spacing value. The second number is how many times it appears among repeating bigram spacings for *all* periods.
It is really quite strange and I can’t explain it.
Here are the top 30 results for Z408;
49: 149
35: 102
61: 86
32: 77
163: 73
74: 68
305: 66
23: 66
62: 62
102: 62
64: 61
128: 61
92: 59
273: 59
91: 58
166: 56
141: 55
97: 53
50: 52
173: 50
159: 50
272: 48
127: 46
28: 45
126: 45
57: 44
243: 44
230: 40
25: 38
196: 38
So, Z408 has a repeating bigram spacing value of 49 that dominates across all periods. So, maybe Z340 being dominated by spacing value 78 is not so significant.
To confirm that Z340 has spacing value of 78 dominating all periods, here are all repeating ngrams at all periods, where the repeats have spacing values of 78:
period 2 ngram +R diff 78 period 2 ngram 2c diff 78 period 3 ngram 5| diff 78 period 5 ngram G# diff 78 period 5 ngram +p diff 78 period 6 ngram F4 diff 78 period 7 ngram 4+ diff 78 period 9 ngram D2 diff 78 period 11 ngram Dc diff 78 period 13 ngram +F diff 78 period 13 ngram F+ diff 78 period 15 ngram R+ diff 78 period 15 ngram pD diff 78 period 16 ngram #+ diff 78 period 16 ngram +5 diff 78 period 17 ngram ++ diff 78 period 18 ngram #R diff 78 period 19 ngram +| diff 78 period 19 ngram |T diff 78 period 19 ngram +4 diff 78 period 20 ngram +D diff 78 period 20 ngram Rp diff 78 period 21 ngram c+ diff 78 period 21 ngram G+ diff 78 period 22 ngram +p diff 78 period 22 ngram 5T diff 78 period 23 ngram 45 diff 78 period 23 ngram GR diff 78 period 23 ngram 2+ diff 78 period 24 ngram p2 diff 78 period 26 ngram 4| diff 78 period 26 ngram pc diff 78 period 26 ngram ++ diff 78 period 26 ngram dG diff 78 period 29 ngram F5 diff 78 period 29 ngram +2 diff 78 period 31 ngram d# diff 78 period 31 ngram +c diff 78 period 32 ngram F| diff 78 period 32 ngram D+ diff 78 period 33 ngram #+ diff 78 period 34 ngram cF diff 78 period 35 ngram RD diff 78 period 36 ngram 2F diff 78 period 37 ngram +D diff 78 period 38 ngram #p diff 78 period 38 ngram G+ diff 78 period 38 ngram +T diff 78 period 39 ngram Tt diff 78 period 40 ngram c4 diff 78 period 42 ngram 24 diff 78 period 42 ngram +5 diff 78 period 43 ngram Gp diff 78 period 44 ngram R2 diff 78 period 45 ngram +| diff 78 period 45 ngram 4T diff 78 period 45 ngram DF diff 78 period 46 ngram Rc diff 78 period 46 ngram +2 diff 78 period 47 ngram c+ diff 78 period 47 ngram d+ diff 78 period 47 ngram p+ diff 78 period 48 ngram +c diff 78 period 49 ngram dR diff 78 period 49 ngram 2+ diff 78 period 51 ngram D4 diff 78 period 51 ngram FT diff 78 period 52 ngram ++ diff 78 period 53 ngram #D diff 78 period 55 ngram t+ diff 78 period 58 ngram GD diff 78 period 58 ngram D+ diff 78 period 58 ngram |t diff 78 period 60 ngram pF diff 78 period 61 ngram 5t diff 78 period 62 ngram #2 diff 78 period 63 ngram tO diff 78 period 63 ngram c5 diff 78 period 64 ngram #c diff 78 period 64 ngram +T diff 78 period 64 ngram d+ diff 78 period 65 ngram +F diff 78 period 65 ngram 25 diff 78 period 66 ngram p4 diff 78 period 66 ngram c| diff 78 period 67 ngram R+ diff 78 period 67 ngram G2 diff 78 period 68 ngram 2| diff 78 period 69 ngram Gc diff 78 period 69 ngram ++ diff 78 period 69 ngram dp diff 78 period 71 ngram +4 diff 78 period 73 ngram p+ diff 78 period 74 ngram D5 diff 78 period 77 ngram D| diff 78 period 77 ngram +t diff 78 period 78 ngram ++ diff 78 period 80 ngram RF diff 78 period 82 ngram +F diff 78 period 84 ngram 4t diff 78 period 84 ngram dD diff 78 period 85 ngram cT diff 78 period 85 ngram #+ diff 78 period 86 ngram R4 diff 78 period 87 ngram 2T diff 78 period 88 ngram +4 diff 78 period 89 ngram p5 diff 78 period 90 ngram Ft diff 78 period 90 ngram G+ diff 78 period 92 ngram p| diff 78 period 93 ngram R+ diff 78 period 93 ngram d2 diff 78 period 94 ngram +5 diff 78 period 95 ngram dc diff 78 period 95 ngram ++ diff 78 period 96 ngram DT diff 78 period 97 ngram +| diff 78 period 98 ngram #F diff 78 period 103 ngram GF diff 78 period 103 ngram +t diff 78 period 104 ngram #4 diff 78 period 107 ngram 5+ diff 78 period 109 ngram G4 diff 78 period 109 ngram R5 diff 78 period 110 ngram td diff 78 period 111 ngram pT diff 78 period 111 ngram #+ diff 78 period 111 ngram +5 diff 78 period 112 ngram R| diff 78 period 113 ngram 4p diff 78 period 114 ngram +| diff 78 period 115 ngram |W diff 78 period 116 ngram G+ diff 78 period 116 ngram +T diff 78 period 116 ngram d+ diff 78 period 124 ngram ct diff 78 period 126 ngram 2t diff 78 period 127 ngram #5 diff 78 period 129 ngram dF diff 78 period 130 ngram #| diff 78 period 131 ngram RT diff 78 period 132 ngram G5 diff 78 period 133 ngram +T diff 78 period 133 ngram Tk diff 78 period 135 ngram t( diff 78 period 135 ngram d4 diff 78 period 135 ngram G| diff 78 period 135 ngram Dt diff 78 period 142 ngram d+ diff 78 period 143 ngram T+ diff 78 period 144 ngram F+ diff 78 period 145 ngram |c diff 78 period 149 ngram #T diff 78 period 150 ngram pt diff 78 period 150 ngram T# diff 78 period 154 ngram GT diff 78 period 155 ngram TG diff 78 period 155 ngram +t diff 78 period 158 ngram d5 diff 78 period 161 ngram d| diff 78 period 170 ngram Rt diff 78
There are 161 of them. The second most frequently occurring spacing value was 35, which occurs 112 times among all periods.
OK I ran another test based on your idea. For all periods (from 1 to 170), I dumped all repeating bigrams and their spacing values (number of positions between repeating bigrams). Then I counted totals of spacing values for all periods. I expected them to be somewhat well-distributed. But the really surprising result is that the spacing value of 78 dominates all the other spacing values. Here are the top 30 results:
78: 161
35: 112
218: 57
71: 52
129: 50
32: 45
226: 43
93: 42
84: 42
162: 42
110: 42
48: 41
75: 39
39: 39
150: 39
210: 35
83: 34
259: 34
137: 34
136: 34The first number is the spacing value. The second number is how many times it appears among repeating bigram spacings for *all* periods.
It is really quite strange and I can’t explain it.
O.k. but the positions that make up all of the period 78 unigram repeats are included with your results.
Take a look at the very first position that is one of a pair of period 78 unigram repeats. Then look at the second position that is one of a pair of period 78 unigram repeats ( but not the same pair as the first ). Find the spacing between them, and that is the period for two identical bigrams spaced at 78 positions.
If you look at it this way, then yes, there are a lot of period x bigram repeats spaced at 78 positions. On the other hand, they could just be a randomly occurring group of symbols distributed about the message that are spaced 78 positions from each other. Together they create a lot of period x bigram repeats, and you are counting them multiple times.
Compare that first position with all of the other period 78 unigram repeat positions, and that’s how many period x bigram repeats that you have spaced at 78 positions. That is why you are getting such a high count. If you check one of the messages that I listed two posts above I am sure that you will find the same thing.
Smokie, that makes sense. The way I’m thinking of it now is: Every period has some number of repeating bigrams, so it’s inevitable that some of them will coincide with the unigram spike at 78. They can coincide with unigram spikes at other widths, but not as often since the spike at 78 is stronger. Is that a fair interpretation?
Smokie, that makes sense. The way I’m thinking of it now is: Every period has some number of repeating bigrams, so it’s inevitable that some of them will coincide with the unigram spike at 78. They can coincide with unigram spikes at other widths, but not as often since the spike at 78 is stronger. Is that a fair interpretation?
I am saying that it is not uncommon for a homophonic message to have a unigram spike, and if there is one, then there will be a lot of period whatever bigram repeats spaced apart from each other at the spike x value because the unigram spike positions create them. Let’s say we have A at 1 and A at 79. And then we have B at 21 and B at 99. The two A’s and the two B’s add to the x = 78 spike for a total count of 4. But they also create a pair of period 20 bigram repeats AB and AB, which are spaced at 78 positions.
I checked the nine messages for these criteria to compare with the period 26 spaced at 78 positions observation:
1. A bigram repeat;
2. Where the period is a divisor of the spacing of the repeat; and
3. The spacing is also the coincidence counting x value.
I probably don’t have enough messages to work with, but didn’t find anything comparable to the 26 / 78 observation. All of the divisors of 78 are represented for the 340, including period 6, 13, 26 and 39 bigram repeats spaced at intervals of 78. I shaded all positive hits with light blue. Some of the messages have a coincidence count x value that have more divisors than others. And I am not sure that counting small periods like 2 or 3 means anything. But so far nothing like the 26 / 78 observation.
Smokie, here are 100 brand new test ciphers you can use, all encoded using normal homophonic substitution. They have the same unigram/bigram/trigram distrbutions as Z340, as well as the same bigram 19 repeats, similar cycles, pivots, "zodiac" signature, etc:
http://pastebin.com/raw/avSEnat1
I will have a more comprehensive presentation of these ciphers soon (including ways to compare their features to Z340).
Thanks for the new messages!
In the 340, there are one period 6, two period 13, four period 26 and one period 39 repeats spaced at 78 positions. So I decided to look at all of them at once and without all of the other symbols.
Period 6: Grey shaded orange bordered 51 and grey shaded blue bordered 53.
Period 13: Orange shaded 19 and grey shaded orange bordered 51.
Period 13: Grey shaded orange bordered 51 and orange shaded 19.
Note that grey shaded orange bordered 51 is common to the period 6 repeat and both period 13 repeats, and the period 13 repeats are actually a period 13 trigram repeat spaced at 78 positions. My symbol 19 is the + by the way.
Period 26: Purple shaded 17 and purple shaded 15.
Period 26: Yellow shaded 5 and yellow shaded 36.
Period 26: Orange shaded 19 and orange shaded 19, which are also common to the period 13 repeats / trigram.
Period 26: Grey shaded blue bordered 53 and blue shaded 11.
Note that the grey shaded blue bordered 53 is common to the period 6 repeat.
Period 39: Green shaded 14 and green shaded 48.
You can see some interesting patterns with these symbols, including one period 19 bigram repeat and one period 19 trigram spaced at 78, which are the only period 19 spaced at 78.
I still don’t know if the coincidence count spike is a rabbit hole or if Bart detected a transposition or other cipher. Bart, maybe you were detecting these symbols when you were talking about 6, 13, 26 and 39?
I checked the period 78 unigram repeat positions for odd and even. There were 15 odd and 20 even.
I have been thinking about polyliteral transposition, where multiple plaintext are transposed in groups, and other ciphers, but don’t have any ideas yet that could create the divisor observation.
So I am wondering what to do next, and at the same time looking forward to another project. I guess we will always wonder about the divisor periods unless we can find another message that is comparable. When I started looking at the spreadsheet messages, I found that some of the evolution messages near the bottom were very similar to each other. So I am thinking that I could find a group of them with the same CC spike. I have not checked the messages that doranchak made available yesterday. I would be happy to check either set of evolution messages to try to find CC spikes and comparable stats if that is what you guys think we should do. Is there a preference between the spreadsheet or yesterday’s messages? Or, if anyone else has an idea for my skill set, that is fine as well.
What do you guys think?
Smokie, it’d be great if you could run tests on my recently posted homophonic substitution ciphers. I’m curious if you can find any features of Z340 that are completely lacking in the generated ciphers. For instance, my hillclimber does not impose the Kasiski unigram spike so I’m wondering if they turn up at all in the generated ciphers.
