Smokie, you actually un-transposed the smokie9 plaintext (meaning that we would never get it assuming you transposed vertically, while you actually un-transposed, as if the the plaintext was transposed already)

Yep, you are correct, there are 2 ways of going about transpositions: "doing" it and "undoing" it. I also have 2 routines, one called "encode_transposition()" and another "decode_transposition()". Naturally, if you call one after another (with the same parameters), you’ll end up with the exact same text you started.

and I had my suspicion because of a corresponding bigram peak at 17 by 19.

Yep, the reason being: if you have a message of length 323, doing/encoding the transposition of width 17 is the same and undoing/decoding the transposition of width 19. Because 232 = 17 x 19. You are just working with the matrix of 17 x 19, and transposing the un-transposed version, or un-transposing the transposed version. I know, it’s confusing, so I just try to remember it. 🙂 The problem is that if the last row is incomplete, i.e. the message is not exactly X * Y symbols long, the results of encoding transposition of width X won’t be the same as decoding transposition of width Y any more. So you pretty much have to test it both ways, at various lengths.

I did an epic run for Z340 once, where I ran all transpositions of width 2 to 22, both directions (encoded and decoded), and all lengths down to 306 (minus 2 last rows) through my solver. I feel I need to redo it again, using 7-grams and with transpositions starting in all 4 corners, just to make sure I covered all bases. 🙂

Well, sorry about the confusion guys. I shuffled the plaintext in smokie9 by moving the plaintext at row 1, column 17 to row 1, column 1. Then at row 2, column 17 to row 1, column 2. Etc. I was trying to simulate the appearance of period 17 bigrams in the first few rows of the 340 that have symbols matching period 18 bigrams in the remainder of the message. I didn’t rearrange the columns. I just worked from the rightmost column to the leftmost column, copying the plaintext while moving from the topmost row to the bottom most row. And I was trying to simulate the low number of period 0 bigram repeats in the bottom half but high number of period 18 bigram repeats in the top half. I showed that transposition is a very plausible way to achieve a low number of bigram repeats by half. Here is smokie10:

The model closely simulates the 340 with regards to those statistics that we have been having trouble with.

However, I tried like hell to get the same results with a high number of period 0 bigram repeats in the first half and a low number of period 0 bigram repeats in the second half. It is somewhat message dependent. But I will report that if I increased the randomization of symbol selection in the second half, like the cycle scores show Zodiac did to some degree, there were always fewer period bigram [EDIT: 0] repeats in the first half and more period 0 bigram repeats in the second half. I clicked on "calculate now" a lot of times, and could not duplicate the 340 in that way.

Thanks a lot for working on it. I feel that the high number of period 18 bigram repeats shows that it is very likely that Zodiac transposed his plaintext before encoding, and will continue to think of ways to detect the shape of the original plaintext message and transposition scheme, as well as models that mimic the 340 stats. It is obviously very difficult.

I also think that an un-transposed version of the 340 that puts period 18 bigram repeats together where resulting period 0 bigram repeat symbols cycle together is a way to approach this. If you have AB repeating two or three times, and AC repeating two or three times, and B cycles with C, then that may be statistically significant. If B and C map to the same plaintext as suggested by cycle scores in the top 1% of all two symbol cycle scores, then you may be finding true bigrams like TH, HE, IN, or whatever. That could help with comparisons between different un-transpositions of the 340. If you have a few to several such examples in one un-transposition, then that, I would think, may be statistically significant.

Smokie

Smokie, I’m thinking that your latest 340 construction hypothesis (a la smokie9 and 10) is on the right track.

Why?

First, the period 19 bigram peak is way to significant to put aside, we have to assume this is it.
Second, your hypothesis has shown that if you don’t get the length of the message exactly right the cipher doesn’t solve.
Third, it’s not hard to imagine that he made a mistake (a letter skipped in the transposition process). Which would then leave the cipher unsolveable (Second) if we don’t include these specific mistakes in our search.

Smokie could you possibly make such a cipher, with just one mistake? And with a known length 323 + 17 filler. For an easy start, thanks.

Jarlve .. Do you mean build a solvable cipher then add in faults to see what makes it fail to decode. Then programmed to overcome such mistakes. I am sure you guys are aware of potential flaws, junk or filler .. Spelling error.. Transposition error.. Numbers..short hand eg + instead of and… Or maybe he used a + sign as a full stop.. or did not use all the letters when writing out the original eg tobeornotobe, a T is missing, technically with no spacings it is not required

Can you guys each put up your best partial solves that you have come up with..it may be interesting to us novices..it may be a starting point if any small match ups occur..

Keep at it guys.. It’s interesting to watch..not sure if it’s the first time such a cohesive group have worked on this project together with understanding and total respect for each other..
Cheers

Smokie, I’m thinking that your latest 340 construction hypothesis (a la smokie9 and 10) is on the right track.

Why?

First, the period 19 bigram peak is way to significant to put aside, we have to assume this is it.
Second, your hypothesis has shown that if you don’t get the length of the message exactly right the cipher doesn’t solve.
Third, it’s not hard to imagine that he made a mistake (a letter skipped in the transposition process). Which would then leave the cipher unsolveable (Second) if we don’t include these specific mistakes in our search.

Smokie could you possibly make such a cipher, with just one mistake? And with a known length 323 + 17 filler. For an easy start, thanks.

Yes of course. I can make smokie11 for you tomorrow. I think that you want me to skip a letter somewhere, but I would like to do exactly as you want. I don’t want you to have to try to solve something and be on the wrong track.

EDIT: I am working on a very easy to follow transposition map that will show mis-alignments of symbols in columns and rows that could be caused by skipping only one plaintext letter during a vertical to horizontal transposition after un-transposing. I think that is what you want. After I make the map and show what happens when one plaintext is skipped as compared to when no plaintext is skipped, then I will make smokie11. I just want to make sure that we are on the same page first. I think that you are thinking about a way to detect that. I will show the maps tomorrow.

2ND EDIT: I think that you are thinking about trying to solve smaller portions of un-transpositions with your program. The portion that occurs before the skipped symbol causes a mis-alignment of symbols in columns and rows.

EDIT: I am working on a very easy to follow transposition map that will show mis-alignments of symbols in columns and rows that could be caused by skipping only one plaintext letter during a vertical to horizontal transposition after un-transposing. I think that is what you want. After I make the map and show what happens when one plaintext is skipped as compared to when no plaintext is skipped, then I will make smokie11. I just want to make sure that we are on the same page first. I think that you are thinking about a way to detect that. I will show the maps tomorrow.

2ND EDIT: I think that you are thinking about trying to solve smaller portions of un-transpositions with your program. The portion that occurs before the skipped symbol causes a mis-alignment of symbols in columns and rows.

Anything of both is fine! We are on the same line. I also wonder if it’s possible to misalign the cipher on purpose (many times, with an extra symbol for instance) and still have a bigram peak at some period. Which we are observing.

Alright, last time there was a lot of confusion, so I am starting out by more clearly stating my transposition scheme based on what I think that Jarlve wants before making smokie11.

Here is an original plaintext tracing map, a transposed plaintext tracing map, and an un-transposed plaintext tracing map. The shape of the message is 17×19. The numbers aren’t symbols in this case. They are positions so that you can see where they wind up after transposition and un-transposition. The shaded numbers are so that you can see what happens after transposition and un-transposition.

This first example has no missing plaintext. There are no mistakes. And the original plaintext tracing map is exactly the same as the un-transposed plaintext tracing map:

This second example has one missing plaintext where I "accidentally" skipped the plaintext at position 162 during transposition. You can see how skipping one plaintext during transposition causes misalignment of a portion of the plaintext after un-transposition.

Jarlve, this is what you want, right? And with all homophones, right?

Yes, thanks allot for your work already. It’s a very clear demonstration of what happens when a mistake is made, and why the 340 possibly is not solving.

1. Smokie11 has the exact same transposition scheme as the second example shown two posts above, except that I skipped a different plaintext instead of the one at position 162. I made a transposition tracing map before doing anything else. The actual message is 323 symbols, or 17×19, as Jarlve requested.

2. The 63 symbols are distributed amongst the letters on the key for the flattest possible symbol frequency (i.e. "e" has a lot of symbols, and "k" doesn’t have very many symbols), resulting in only 54 period 18 bigram repeats. That was the highest number I could get in as many as twenty tries. All are homophones. Randomization of symbol selection from cycles increases as you go from the top rows to bottom rows.

3. The plaintext is from Jarlve’s 100 plaintext library, which I might add is quite handy for making messages.

4. There are 31 "apparent" period 0 bigram repeats in the entire message. There are 0 "apparent" period 0 bigram repeats in the first half, and only four "apparent" period 0 bigram repeats in the second half. To accomplish this took *almost* no effort whatsoever.

5. I did not try to solve it, and I suppose it may be very easy to solve except for the areas around the vertical misalignment line. We can make an adjustment so that smokie12 is a little bit more difficult if necessary.

18 9 15 5 27 11 41 51 58 16 2 31 21 62 30 37 1
43 4 13 7 17 8 32 36 29 44 49 11 19 24 14 10 40
20 35 12 45 33 54 5 57 50 53 61 28 59 1 42 26 2
9 47 52 3 46 24 4 30 8 55 27 29 26 7 42 2 63
13 40 31 18 37 4 16 28 41 26 49 51 44 32 11 45 23
58 60 50 39 57 59 4 54 6 61 34 13 6 52 62 53 19
59 26 8 63 7 14 38 48 32 1 20 21 15 24 33 18 29
6 42 35 45 37 54 28 4 46 7 40 26 12 2 55 49 19
28 27 6 4 26 50 57 60 13 50 4 32 34 41 24 56 61
17 31 45 40 54 12 26 52 4 9 29 51 30 43 13 25 44
36 58 26 53 50 63 1 24 11 7 62 3 48 29 23 42 53
19 9 6 7 4 39 25 56 26 31 46 30 15 40 20 3 18
23 52 47 16 44 4 43 33 17 5 13 7 25 35 34 6 8
50 36 48 39 53 37 1 59 12 31 27 29 24 26 28 56 20
51 49 58 40 57 17 6 48 4 47 62 44 14 61 42 16 45
59 63 7 26 35 22 30 28 4 46 13 2 52 50 18 45 37
60 43 8 48 26 32 49 46 4 45 19 54 1 20 57 13 56
33 6 23 12 32 7 53 24 59 39 5 46 10 36 28 8 45
40 47 26 27 36 1 9 14 16 6 49 63 17 42 3 18 15
1 44 6 56 37 60 14 4 6 30 35 12 24 54 26 4 41

EDIT: I un-transposed and tried to solve with ZKD v1.0 Beta 4. It did not solve. Then I moved the symbols to the left of the misalignment line from row 19 to row 1 and moved the symbols on rows 2-18 and to the left of the misalignment line down one row. The message solved after doing that. I also thought of one semi-pragmatic way to try to solve the message not knowing where the misalignment line is or on what row it is staggered. Tried it and it works not too bad. Not a perfect solve, but there are a lot of actual words and short sentence fragments so far. Will discuss after Jarlve and anyone else want to try it.

EDIT 2: I am also thinking of another possible way to try to find a vertical misalignment line, similar to the concept of comparing lists of period 17, 18 and 19 bigrams to identify how long the message is. Maybe comparing lists of period 35 bigrams as found in sets of three contiguous columns with period 18 bigrams in the remaining columns to see if there are a "lot" of them with the same symbols. That would take some time for me, and I am not sure if it will work. But I will put that on my to do list.

EDIT 3: Jarlve, could you please run your hot-spot program on smokie11? I am curious to see what that will show, if anything.

Thanks smokie,

Here is the hotspot map after transposition of the 17 by 19 block, so it’s a misaligned solve. Please take note that the last 5-6 squares are not normalized and will be darker.

I solved the message by filling all possible skips (at positions). I’m up for something harder, maybe a couple of skips or if you have any ideas. Could also consider the opposite!

157

Score: 23401 Ioc: 660 M: 198 C: 323 S: 64

gandalfstoppedont
hebridgestandingi
nthemiddleofthesp
anallowingtheothe
rstoescareheleane
donthestaffinhisl
efthandandheldthe
swordglamdringgle
amingcoldandwhite
inhisrightthebalr
ogsteppedontotheb
ridgefacinggandal
fandtheshadowabou
titreachedoutlike
twocastwingsitrai
seditswhipandthet
hongswhinedandcra
ckedfirecamefromi
tsnostrilsbutgand

<!-$!L&D]W/BI$A-2
(4"O[$'Z1Q8-:>V')
MYT^,K$:?3A&Q(%1/
H-RL?AGUM<2TSWY(%
OX2.;19Ra;=^L;R@;
:AM]=%X_H&&U@=UD+
;&EJ8-:!V$J3+:]=I
D7:'?8,:C[M'<LS
!,KM<9N+:RM:7J[QZ
)@([XC>'TYE(Z"!+0
.'P_4B/^$V__J3"
0U:'S&H#)-<'R-:8?
&8@:Q(4P=R$.GH".5
Q[QCZR#=S$NFY?)*^
_GN6HD27>@'P[2OH>
1%$>YP7JKBH@$ET4E
JNM<1GTU-I:!V$#C!
#*3$&UO398,4&0N,K
2DVNPE0)LX"5]<8M$

I also thought of one semi-pragmatic way to try to solve the message not knowing where the misalignment line is or on what row it is staggered. Tried it and it works not too bad. Not a perfect solve, but there are a lot of actual words and short sentence fragments so far. Will discuss after Jarlve and anyone else want to try it.

Sure, I’m all ears.

I just tried it on the 340 and got the best response going from west to east, and speaking of coincidence it also peaked at position 157 just as the smokie11.

157

Score: 18588 Ioc: 739 M: 198 C: 323 S: 64

mhimapolinearpurs
ueherandihodrunes
tsnottogottheneac
nditandfoundsoisa
catchrasbraannedt
oprailsintereitha
ulshesthourtorrhf
orshadrshareeitsr
andturnhuleethers
andicheinnisharpc
asormisthenthehot
hemitedrachmandre
dingupatcsitethei
relitthecreosonan
tschoriesitthepaa
recoandatioteisle
factienceandishme
diusesonargueatri
ceofftheislegcoup

H+M8|CV@KUcy5C^W(
EB+*5k.L-+l#2E.B)
R(UVFFz9zRR+4>f|p
>#Z3P>Ldl/JNbVM)|
pOGp+2|<a5||.UqLF
l%WO&D(MUt*5cZG+k
^D(+4(G+VE5FV52+d
VW)+k#2b+|TB4-R)W
PYLR/5J+^D4ct+cW<
k.#Kp+fZJYM(+|TC7
|<z28KjR+B.;+c+zt
1*H_Rq#2Op+8y.LWB
LKJ9^%OFpb&RB31c_
TfSMF;+B7TBlXz6PY
G)p+l2_c<MFG1BCOO
2BpzOUNyFKzF*K<SB
dy7t-cYABO6N:(+H*
N:^j*Xz6y29^4OFT-
pclddG+4-<Sf9pl/C

150

Score: 18553 Ioc: 690 M: 198 C: 323 S: 64

mhimabovenearbuss
ueherandihadtunes
tsnofforitthereac
rditardloundooisa
cotchtalaraannedf
ansongsintereitha
ugshesthourforthl
osshadtohareeitss
andturnhugeethesl
andicheinnisharby
alotmisthenthehot
hemptedtochmandse
dinrunofcontethep
reliftheyreadoran
tschatpeliftheboo
tecoondafiofeille
laytienceordishme
diusedonatrueofri
ceallthrillercaub

H+M8|CV@4Ucy5C^W(
EB+*5k.LK+l#2E.B)
R(UVFFz9-RR+4>f|p
>#Z3P>Ldz/JNbVM)|
pOGp+2|<l5||.UqLF
l%WO&D(MUt*5cZG+k
^D(+4(G+VE5FV52+d
VW)+k#2b+|TB4-R)W
PYLR/5J+^D4ct+cW<
k.#Kp+fZJYM(+|TC7
|<z28KjR+B.;+c+zt
1*H_Rq#2Op+8y.LWB
LKJ9^%OFpb&RB31c_
TfSMF;+B7TBlXz6PY
G)p+l2_c<MFG1BCOO
2BpzOUNyFKzF*K<SB
dy7t-cYaBO6N:(+H*
N:^j*XzAy29^4OFT-
pclddG+6-<Sf9pl/C

My idea for solving was:

1. Un-transpose;
2. For each possible vertical misalignment line

a. Move the symbols in row 19 and to the left of the line up to row 1. Then move the symbols in rows 2-18 and to the left of the line down one row;
b. Because the line is staggered on some unknown row, expand all symbols in the columns to the left of the line and to the right of the line;
c. Try solving to detect where the line is;
d. Take the highest scoring or best solution and then use trial and error working row by row to find where the line is staggered.

A rough concept that may be more efficient, considering all of the possible 340 message lengths, shapes and transposition directions. But I will leave that to your judgment.

In general, I want to move forward with constructing messages that simulate actually detected 340 attributes that are of statistical significance, rather than taking a trial and error approach with a concept with no detection before hand. However, your idea that Zodiac may have skipped one or more plaintext while transposing is very plausible yet probably very difficult to detect. Transposition is difficult enough for me to do it with a spreadsheet and keep everything straight, much less with a pen and paper. I want to continue to pursue the period 18 bigrams and idea that Zodiac may have skipped or added plaintext during vertical transposition of an otherwise rectangular shaped message.

What happened when you tried to solve smokie11 after un-transposition but before filling all possible skips?

When you tried filling in one missing plaintext on all possible positions of the 340, did you assume a message of 323 symbols long, or did you vary your message length?

I want my messages from now on to be very precisely constructed so that you or I or both of us or anyone else can test one simple, additional idea at a time and work on detection and trial and error techniques. Or pursuant to your specific instructions. If you want simple, additional ideas to be a surprise so that you feel more challenged, then that is o.k. as well.

I will make smokie12 for you soon, after I post more information about smokie11. I have some other stuff to do today also. I need to run all of the gasoline through my lawnmower because winter is coming among other things.

Well, I had to know if the hot-spot program may help us detect a misalignment due to skipping a plaintext during vertical to horizontal transposition. It seems that it does not.

Here is the smokie11 transposition tracing map:

And here is the key:

A 1 24 40 50
B 2
C 3 25
D 4 26
E 5 27 41 51 58 62 19 20
F 6
G 7 28
H 8 29 42 52
I 9 30 43 53 59
J
K 10
L 11 31 44
M 12
N 13 32 45 54
O 14 33 46 55 60
P 15 34
Q
R 16 35 47
S 17 36 48 56
T 18 37 49 57 61 63
U 21 38
V 22
W 23 39
X
Y
Z

A key with this type of construction for minimizing symbol frequency is going to result in a minimum number of period 18 bigram repeats. There were 54 period 18 bigram repeats ( I count AB AB AB as three total ). The 340 has 72 by my count, implying that the key was much "flatter."

I have the transposition tracing map for smokie12 done, and am ready to start encoding.

Copying two questions from the post above:

What happened when you tried to solve smokie11 after un-transposition but before filling all possible skips?

When you tried filling in one missing plaintext on all possible positions of the 340, did you assume a message of 323 symbols long, or did you vary your message length?

A rough concept that may be more efficient, considering all of the possible 340 message lengths, shapes and transposition directions. But I will leave that to your judgment.

Expanding symbols will require much iterations and introduces a factor of uncertainty. I’ll keep the current method.

What happened when you tried to solve smokie11 after un-transposition but before filling all possible skips?

That’s the solve you see in the hot spot image, it scored considerably higher and some readable fragments can be found in there. The 340 does not produce such a response in any way.

When you tried filling in one missing plaintext on all possible positions of the 340, did you assume a message of 323 symbols long, or did you vary your message length?

Assumed 323. Perhaps you could make a cipher with a different length and one skip (both length and skip position unknown to me). By the way, the period is equal to the distance between the bigram. So we’re talking about a period of 19 here.

I have a question for you. I count 25 bigrams repeats for the 340 in normal orientation and 29 when mirrored. 37 period 19 bigrams normal orientation and 39 period 15 bigrams. Should we consider the stronger response from the mirrored version?