I agree. For what it is worth, my heatmap shows that deleting symbols in the area of the pivots would increase the count of period 30 repeats.

One idea would be to first explore the idea of misalignments without homophonic encoding. Just transpose plaintext messages, add or delete symbols in certain areas and with different configurations, and see just how unreadable the plaintext is when untransposed. Experiment a little and see just how many skips or nulls it would take to make an untransposed plaintext message unreadable.

As I said in the other thread, maybe the pivots are an intentionally planted clue. That would be very Zodiac like. Maybe he put the pivots in the message first before all of the other transposed plaintext. And he just transcribed around the pivots. I have a busy schedule this week, but would be interested in seeing if untransposed plaintext is readable using that scheme. I can play around with it maybe later this week or on the weekend.

Jarlve, you might also want to check out my experiment with smokie38 where I tried to transcribe around column 9 to create period 29 repeats:

About the 5th post down, here: viewtopic.php?f=81&t=2617&start=1370

Step 1: Transpose plaintext. The inscription and transcription rectangles are both 17 x 20 to keep things really simple. I inscribed left right top bottom, lifted the plaintext from the inscription rectangle top bottom left right, and transcribed left right top bottom. Scroll to the right to see the transposed plaintext.

Step 2: Re-draft the message around the pivots. Scroll to the right to see the transposed plaintext written around the pivots.

Step 3. Untranspose as if the pivots were not there. On the far right is the untransposed message outlined in blue, which could not be solved. Scroll to the right to see.

Whether he did this or not, it is a good idea for a route cipher. The message sender and receiver could agree that certain positions in the message would be nulls, and the count and positions of the nulls could change with each subsequent message. At least one of the cryptography books of the era discussed using nulls to make solving the message very difficult. I will look for it.

EDIT:

I L I K E K I L * L H C T T H R I
E B E C A U S E Y Y H E C E I T I
H F U N I T I S L * E A H A N G E
A N K I L L I * I O A R K S O F F
I N T H E F O N M I L N B E S T P
S E M A N I S R G T A E T W H E N
G E R O U S A T R O O V E B O R N
T O K I L L S N H W U U D A L L T
V E S M E T H O I E N W D W I L L
L I N G E X P E M E I G E S I W I
E V E N B E T E M M R * M Y N A M
T I N G Y O U T R * I * I L L T R
I T H A G I R R E * S * P E O P L
R T O F I T I L R * M S S O M U C
D I E I W I L S T E A E F U N T H
N P A R A D I L T O T L D G A M E
E I H A V E K C B E * T B E C A U
E C O M E M Y I * R S O S T D A N
L N O T G I V S E O N L O F A L L
B E C A U S E E L * T H I N G G I

I agree. For what it is worth, my heatmap shows that deleting symbols in the area of the pivots would increase the count of period 30 repeats.

I’m actually adding symbols – increasing the cipher length. Because I thought that for period 30 to shift to 29, some characters had to be missing. Here are two graphs, the x-axis is the first symbol added from position 1 to 341 and the y-axis is the second symbol added from position 1 to 342. The second graph is the same but here symbols are removed. The color intensity is the bigram score. The 340 was mirrored prior to any other operation and the last operation is period 30. So it compares adding or removing symbols and its effect on period 30 bigram repeats.

– Adding symbols is superior to removing symbols in boosting period 30 bigram repeats, though removing 2 symbols may play a small part in that.
– Both graphs show somewhat similar distributions, peaking around position x: 200, y: 200 (give or take) very close to the pivots.
– The highest bigram return of both graphs is "28 (Add character(210), Add character(215), Period(UTP,30))", very close to the positions with increased AZdecrypt scores.

Symbols added:

Combinations processed: 116622/116622
Measurements:
- Summed: 2110746
- Average: 18.09903791737408
- Lowest: 10 (Add character(151), Add character(272), Period(UTP,30))
- Highest: 28 (Add character(210), Add character(215), Period(UTP,30))

Symbols removed:

Combinations processed: 115260/115260
Measurements:
- Summed: 2050036
- Average: 17.78618774943606
- Lowest: 11 (Remove character(31), Remove character(110), Period(UTP,30))
- Highest: 25 (Remove character(1), Remove character(241), Period(UTP,30))

As I said in the other thread, maybe the pivots are an intentionally planted clue. That would be very Zodiac like. Maybe he put the pivots in the message first before all of the other transposed plaintext. And he just transcribed around the pivots. I have a busy schedule this week, but would be interested in seeing if untransposed plaintext is readable using that scheme. I can play around with it maybe later this week or on the weekend.

Good idea but how does that explain the observation that the pivots are made up of period 29 repeats? It’s doranchak who made that observation right?

I highly recommend reading The Department of the Army Technical Manual ( published 1955 ), pages 18 – 26 here:

https://www.nsa.gov/news-features/decla … 078809.pdf

From page 22:

Nulls, when employed for the purpose of
making cryptanalysis more difficult, may also be inserted in specific positions
as prearranged, or they may be inserted at random if the system
permits.

As I said in the other thread, maybe the pivots are an intentionally planted clue. That would be very Zodiac like. Maybe he put the pivots in the message first before all of the other transposed plaintext. And he just transcribed around the pivots. I have a busy schedule this week, but would be interested in seeing if untransposed plaintext is readable using that scheme. I can play around with it maybe later this week or on the weekend.

Good idea but how does that explain the observation that the pivots are made up of period 29 repeats? It’s doranchak who made that observation right?

I believe that I made the observation. But you are right. If he put the pivots there first at period 29, it would have been quite a coincidence.

Jarlve, you might also want to check out my experiment with smokie38 where I tried to transcribe around column 9 to create period 29 repeats:

About the 5th post down, here: viewtopic.php?f=81&t=2617&start=1370

I’m mindblown and need some time with this, it looks very promising. I applied your transposition to a plaintext and then mirrored it and a few things stood out, bigram peaks at period 15 and 31. It managed to shift the period from 30 to 31. Can you do it the other way around? Reading from your post it seems that you have not found strong evidence of a column like this in the 340 but could you see how it looks if you add columns from position 1 to 18 instead of removing? And also for rows?

Here’s your transposition matrix with spaces added for easier visual reference. The middle column does not follow the transposition direction and seems to create a misalignment that causes period 2 repeats to shift.

252 234 216 198 180 162 144 126 270 108 90  72  54  36  18  323 305
287 269 251 233 215 197 179 161 288 143 125 107 89  71  53  35  17
322 304 286 268 250 232 214 196 306 178 160 142 124 106 88  70  52
34  16  321 303 285 267 249 231 324 213 195 177 159 141 123 105 87
69  51  33  15  320 302 284 266 325 248 230 212 194 176 158 140 122
104 86  68  50  32  14  319 301 326 283 265 247 229 211 193 175 157
139 121 103 85  67  49  31  13  327 318 300 282 264 246 228 210 192
174 156 138 120 102 84  66  48  328 30  12  317 299 281 263 245 227
209 191 173 155 137 119 101 83  329 65  47  29  11  316 298 280 262
244 226 208 190 172 154 136 118 330 100 82  64  46  28  10  315 297
279 261 243 225 207 189 171 153 331 135 117 99  81  63  45  27  9
314 296 278 260 242 224 206 188 332 170 152 134 116 98  80  62  44
26  8   313 295 277 259 241 223 333 205 187 169 151 133 115 97  79
61  43  25  7   312 294 276 258 334 240 222 204 186 168 150 132 114
96  78  60  42  24  6   311 293 335 275 257 239 221 203 185 167 149
131 113 95  77  59  41  23  5   336 310 292 274 256 238 220 202 184
166 148 130 112 94  76  58  40  337 22  4   309 291 273 255 237 219
201 183 165 147 129 111 93  75  338 57  39  21  3   308 290 272 254
236 218 200 182 164 146 128 110 339 92  74  56  38  20  2   307 289
271 253 235 217 199 181 163 145 340 127 109 91  73  55  37  19  1

Wow, that is an interesting find! I wish I had more time to look closer all those new ideas. Unfortunately I am still working on a command line version of a solver which costs a lot of my spare time.

At least one of the cryptography books of the era discussed using nulls to make solving the message very difficult. I will look for it.

I think it was this book:
Codes and Ciphers – A History of Cryptography from Alexander D’Agapeyeff

I’m mindblown and need some time with this, it looks very promising. I applied your transposition to a plaintext and then mirrored it and a few things stood out, bigram peaks at period 15 and 31. It managed to shift the period from 30 to 31. Can you do it the other way around? Reading from your post it seems that you have not found strong evidence of a column like this in the 340 but could you see how it looks if you add columns from position 1 to 18 instead of removing? And also for rows?

Here’s your transposition matrix with spaces added for easier visual reference.

I am glad that you liked smokie38. However it did not work to create 340 bigram repeat stats because the center column caused too much distortion to the transposition structure. It may have been more successful if only in the bottom half, but I felt that would have been much more arbitrary than one full gibberish column.

You should have gotten a spike at period 29 when mirroring, so maybe try some other plaintexts. They are not all the same. I used the plaintext with the highest count of period 2 bigram repeats as compared to period 1 repeats, and I was still unable to reproduce 340 period 29 stats.

I was very pleased with the shuffle test. I can shuffle a region of any size or shape, and the test works well to identify gibberish rows and columns. I have not tried it on other types of regions. It would probably be more reliable when taking a slice through the transposition structure. I did not find any evidence that column 9 of the 340 is independent of the assumed transposition structure.

Not sure what you mean by adding columns from 1 to 18.

One approach to the pivots would be to try to find a transposition scheme that would naturally create them, whether single or in pairs, AND the period 15 / 19 repeats. I think that more extensive work in that area would help us to decide what the pivots really are. Natural result of transposition, intentional visual clue installed before or after drafting the balance of the message, or statistical anomaly.

I am working on that and considering, among other things, multiple inscription or transcription rectangles, incomplete rectangles, and alternating columns or rows during inscription or transcription.

EDIT: I really like the tracing map, or "matrix" idea because it allows us to easily share transposition ideas. I saw "matrix" in your new program. Is that what it does?

I am glad that you liked smokie38. However it did not work to create 340 bigram repeat stats because the center column caused too much distortion to the transposition structure. It may have been more successful if only in the bottom half, but I felt that would have been much more arbitrary than one full gibberish column.

I agree that it causes to much distortion. A misalignment caused by either addition or omission of characters is probably be going to minimal unless it somehow chimes in with the period itself. Given the results from the main post I’d say there’s a good chance that it exists. Though, a minimal misalignment would probably not prevent us from solving the cipher straight away. So I feel there’s something else also. Which could be another misalignment, but not in the form of addition or omission.

An example could be the following strange observation (I have this shared before). Right shift all the columns of the 340 by 1 and measure period 19 and you’ll now find 45 bigrams. Going from 37 to 45 with such a minimal step blows my mind, the statistical difference between 37 and 45 is huge and there is no combination of both column and row offsets in either the normal/mirror/flipped/reversed period 15/19 variations that scores higher. And it is just one step away. I need to do more research on this if it could be a step in the process or not.

dHER>pl^VPk|1LTG2
)Np+B(#O%DWY.<*Kf
JBy:cM+UZGW()L#zH
2Spp7^l8*V3pO++RK
/_9M+ztjd|5FP+&4k
(p8R^FlO-*dCkF>2D
|#5+Kq%;2UcXGV.zL
9(G2Jfj#O+_NYz+@L
Kd<M+b+ZR2FBcyA64
--zlUV+^J+Op7<FBy
OU+R/5tE|DYBpbTMK
F2<clRJ|*5T4M.+&B
Rz69Sy#+N|5FBc(;8
+lGFN^f524b.cV4t+
+yBX1*:49CE>VUZ5-
2|c.3zBK(Op^.fMqG
LRcT+L16C<+FlWB|)
p++)WCzWcPOSHT/()
c|FkdW<7tB_YOB*-C
+>MDHNpkSzZO8A|K;

17 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
33 18 5 19 20 21 22 23 24 25 26 27 28 29 30 31 32
41 20 34 35 36 37 19 38 39 15 26 21 33 13 22 40 1
16 42 5 5 43 7 6 44 30 8 45 5 23 19 19 3 31
54 46 47 37 19 40 48 49 17 11 50 51 9 19 52 53 10
21 5 44 3 7 51 6 23 55 30 17 56 10 51 4 16 25
11 22 50 19 31 57 24 58 16 38 36 59 15 8 28 40 13
47 21 15 16 41 32 49 22 23 19 46 18 27 40 19 60 13
31 17 29 37 19 61 19 39 3 16 51 20 36 34 62 63 53
55 55 40 6 38 8 19 7 41 19 23 5 43 29 51 20 34
23 38 19 3 54 50 48 2 11 25 27 20 5 61 14 37 31
51 16 29 36 6 3 41 11 30 50 14 53 37 28 19 52 20
3 40 63 47 42 34 22 19 18 11 50 51 20 36 21 58 44
19 6 15 51 18 7 32 50 16 53 61 28 36 8 53 48 19
19 34 20 59 12 30 35 53 47 56 2 4 8 38 39 50 55
16 11 36 28 45 40 20 31 21 23 5 7 28 32 37 57 15
13 3 36 14 19 13 12 63 56 29 19 51 6 26 20 11 33
5 19 19 33 26 56 40 26 36 9 23 42 1 14 54 21 33
36 11 51 10 17 26 29 43 48 20 46 27 23 20 30 55 56
19 4 37 25 1 18 5 10 42 40 39 23 44 62 11 31 58

You should have gotten a spike at period 29 when mirroring, so maybe try some other plaintexts. They are not all the same. I used the plaintext with the highest count of period 2 bigram repeats as compared to period 1 repeats, and I was still unable to reproduce 340 period 29 stats.

I noticed the variability, something to take in mind with the 340.

Not sure what you mean by adding columns from 1 to 18.

A misunderstanding on my part, you shuffled the symbols of these columns.

EDIT: I really like the tracing map, or "matrix" idea because it allows us to easily share transposition ideas. I saw "matrix" in your new program. Is that what it does?

Yes. Put your cipher, be it numerical or characters in the input window and put a transposition matrix into the output window and then go to functions -> transposition and click on "use transposition matrix" and then transpose/untranspose. One way to generate these from scratch in AZdecrypt is to go to text filters and click on generate numbers and fill in the from# and to# fields and click on process. After that you can use dimension and transposition to define it.

This is a follow up to the main post. Now, 2 new unique symbols were added with the mirored 340 as a starting point. The x-axis are the positions ranging from 1 to 341 were the first symbol was added and the y-axis the second symbol from 1 to 342. The resulting variation was then untransposed with period 15 and lastly normal and reversed strings are generated. The square on the left is the normal string and the square on the right is the reversed. Thus, each pixel is a solve and its color intensity is the AZdecrypt score with 10 restarts.

LBEHERERSTOOBREAC
URSUEINSHOWADOWAS
STEFORNOHITSETTOA
LLEHALARARONEARDI
NREGIONEDINCINCAR
SASOUSISACUITCOLL
UCINGSOOTITUPRESE
TISSGUESHIHAREATH
LEASINTHANDOFASAI
NBRETHETERIAPOFTH
ESOFOURDOUTOFGALS
INCOMSFIRESPECTAN
DASOUTHEDIAINTSIN
COASFRCHNONTHATWA
NISTHINGITHSETOAM
USINTHENHERWIFEAS
PEARTSETHNARCHCLO
THEREDSSARINGARSI
NSONATHADINALLITS
OFFICEHHAROTHEGRO
DW


Combinations processed: 233244/233244
Measurements:
- Summed: 4698243026.503229
- Average: 20143.03916286476
- Lowest: 19476.80753592638 (Add character(74), Add character(130), Period(UTP,15), Reverse)
- Highest: 21341.11905763229 (Add character(200), Add character(331), Period(UTP,15), Normal)

Observations,
– There is a cross of increased scores, which sits roughly at x: 200, y: 200. In line with previous observation. If more characters were omitted, why aren’t they showing up as much?
– The area of increased scores is hardly visible with the reversed string, indicating that one direction is better than another (plaintext phenomenon).
– Highest scoring variation: 21341.11905763229 (Add character(200), Add character(331), Period(UTP,15), Normal). The highest 340 score I’ve seen. Though 2 unique symbols have been added raising the multiplicity slightly and 233244 combinations were processed.
– Some interesting fragments in the resulting plaintext "INALLITSOFFICE" and "COLLUCING".

A good hypothesis is that the cipher once existed in a 23 by 15 grid and then was taken off by columns and transcribed in a 17 by 20 grid bottom-to-top or right-to-left and 5 characters (or less?) had to be left out (going from 345 to 340). It seems hard to reconcile leaving out 5 characters with bigram findings though. But it makes allot of sense for everything else. I made a cipher as such but had to raise the raw ioc (repeat potential) significantly above that of the 340 to match 41 bigram repeats for mirrored period 15. I left out 5 characters. Can you solve it?

%H:G)9?;AWQ^8VT+"
?9=8-;,=;5=Y=A94P
@,'>.+F(OG1ST$H7=
U]8/9*LZPT=Q^Q4=C
%,^VJ9I"A)7VR;GA!
)'JV;.2>TX=H*CSK5
N;H;HV9PTOT,G:A]Q
S,A<D7VUR_E@QVF+G
:.%^TVI_EV>";?=18
RLV:;A,GH8,]PW2G9
J5(A4GT)VPH]=9$;H
,6+G?R5BO#FA9<T=D
!J5=TV]Q'0/*(XH8;
A;2KGS&7_R%"9UI@E
9'^=1@>K)AQVKVVVH
B)TVPU:<;O=H;HC,
]GH.LG"F9;^FH8PA4
2R@>9A;/4$V8:V2+M
4PVK3:!9#AS^;FDU7
I[^ETJC"F,7A%Q8?H

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
7 6 18 13 19 8 20 18 8 21 18 22 18 9 6 23 24
25 20 26 27 28 16 29 30 31 4 32 33 15 34 2 35 18
36 37 13 38 6 39 40 41 24 15 18 11 12 11 23 18 42
1 20 12 14 43 6 44 17 9 5 35 14 45 8 4 9 46
5 26 43 14 8 28 47 27 15 48 18 2 39 42 33 49 21
50 8 2 8 2 14 6 24 15 31 15 20 4 3 9 37 11
33 20 9 51 52 35 14 36 45 53 54 25 11 14 29 16 4
3 28 1 12 15 14 44 53 54 14 27 17 8 7 18 32 13
45 40 14 3 8 9 20 4 2 13 20 37 24 10 47 4 6
43 21 30 9 23 4 15 5 14 24 2 37 18 6 34 8 2
20 55 16 4 7 45 21 56 31 57 29 9 6 51 15 18 52
46 43 21 18 15 14 37 11 26 58 38 39 30 48 2 13 8
9 8 47 49 4 33 59 35 53 45 1 17 6 36 44 25 54
6 26 12 18 32 25 27 49 5 9 11 14 49 14 14 14 2
56 5 15 14 24 36 3 51 8 31 18 2 8 2 42 20 60
37 4 2 28 40 4 17 29 6 8 12 29 2 13 24 9 23
47 45 25 27 6 9 8 38 23 34 14 13 3 14 47 16 61
23 24 14 49 62 3 46 6 57 9 33 12 8 29 52 36 35
44 63 12 54 15 43 42 17 29 20 35 9 1 11 13 7 2

I belief to have found evidence of transposition misalignment in the 340.

Here’s a graph that shows it. The x-axis are the positions from 1 to 341 at which a new unique symbol was added. And the y-axis is the AZdecrypt score with 100 restarts. Prior to the other operations the 340 was mirrored. Highest score is slightly over 21000 with a symbol added at position 200.

That’s very interesting. It is similar to the spikes I found a while back when I was playing around with different transposition schemes:

viewtopic.php?p=43702#p43702

The noteworthy result from those experiments was that several high scoring AZDecrypt runs were due to operations involving "PeriodColumn(2) Period(18)" operations, which produced more repeating bigrams. Generally, AZdecrypt seems to score higher whenever repeating bigrams are high. Many combinations of operations are able to produce high numbers of repeating bigrams, so it’s hard to figure out which combination is correct.

To recap those operations, PeriodColumn(2) simply re-arranges the columns 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 as 1,3,5,7,9,11,13,15,17,2,4,6,8,10,12,14,16. Then, Period(18) simply untransposes the entire cipher with period 18.

I am wondering if the various ways of getting AZDecrypt spikes at around 21000 are indications of being close to a real untransposition, or if those operations are truly false positives. If I recall correctly, an AZdecrypt spike of around 21000 is only about 1 sigma (1 standard deviation) away from the average of AZdecrypt runs on random shuffles of Z340.

An example could be the following strange observation (I have this shared before). Right shift all the columns of the 340 by 1 and measure period 19 and you’ll now find 45 bigrams. Going from 37 to 45 with such a minimal step blows my mind, the statistical difference between 37 and 45 is huge and there is no combination of both column and row offsets in either the normal/mirror/flipped/reversed period 15/19 variations that scores higher. And it is just one step away. I need to do more research on this if it could be a step in the process or not.

As a point of reference, the "PeriodColumn(2) Period(18)" operation I mentioned produces 44 bigrams. It’s interesting that we can find simple steps that yield significant increases in ngrams, with what seems to be a wide variety of steps.

Simple modifications allow more bigrams to be produced:

"PeriodColumn(2) Period(18) Swap(5, 105, 6, 1) Swap(101, 225, 3, 1)" produces 53 bigrams. Swap(5,105,6,1) exchanges a block of height 6 width 1 at position 5 with the same sized block at position 105. Swap(101,225,3,1) exchanges a block of height 3 width 1 at position 101 with the same sized block at position 225.

"PeriodColumn(2) Period(18) Swap(4, 104, 10, 1) SwapLinear(101, 238, 55)" produces 52 bigrams. SwapLinear(101, 238, 55) exchanges a substring of length 55 at position 101 with a substring of the same length at position 238.

"PeriodColumn(2) Period(18) RectangularSelection(7, 0, 7, 3) Period(10)" produces 52 bigrams. Period(10) is only applied to the RectangularSelection(7,0,7,3) which means a rectangular selection with upper left corner at row 7 column 0, and having height 7 and width 3.

There are several other variations of this type but many seem to have the "PeriodColumn(2) Period(18)" in common.