Test, log all unique 2-symbol cycle break positions, heat maps:

Z340:

Z408:

Test, log all unique 2-symbol cycle break positions for symbols that have a similar frequency (max -2 or +2), heat maps:

Z340:

Z408:

The previous maps had the problem that the real sub set of cycles is lost in the other 100’s or 1000’s of cycles. I realized that I could use AZdecrypt’s Merge sequential homophones solver to come up with an assumed set of "real/best fit" cycles. And measure the breaks from there.

Results look much better. This is getting somewhere.

Still some problems:

1) Assumed set of cycles.
2) My measuring of breaks perhaps not optimal (pondering).
3) Merge sequential homophones will try to avoid bad cycles which may nevertheless be real.

Z340:

Z408:

Some newer renditions, not sure if better or worse, pretty similar in all:

Z340:

Z408:

Z408 with added 18th column consisting of random symbols, does not show up on map but has more breaks:

This is really interesting! Thanks for sharing.

So – let me check I’m following along – the heatmaps are showing symbol positions where mostly perfect 2-symbol ciphers are broken (i.e. if a cycle is FBFBFBFBB*FBFB, it is the B* which is highlighted)?

In this case, it would be the regions between highlighted squares which would be the break "zones" as I called them, which we would (maybe) be expecting to overlap with the 18th column, if it was removed after homophonic encipherment?

As Dave explained much better than me, once you replied to my initial idea it led me to agree – there are so many plausible 2-symbol cycles would this test be easier if we limited it to minimum 3 or 4-symbol cycles (i.e. FBPYFBPYFBPY, rather than FBFBFBFB)?

As an aside, I had a question on this test:

Ran another test with a perfect 2-symbol cycles measurement, it is much in favor of the Z340 not having a 18th column after sequential homophonic substitution:

what are 9: and 18: referring to, the change in cycle score before/after removing the 18th column? Is the problem with this test for the 340 not that, unless we perfectly fill the 18th column as it was originally we would expect a random column to do to the cycle repetitions the same as the absence of that column, therefore we would expect this result even if your 18th column was removed post encipherment?

So – let me check I’m following along – the heatmaps are showing symbol positions where mostly perfect 2-symbol ciphers are broken (i.e. if a cycle is FBFBFBFBB*FBFB, it is the B* which is highlighted)?

Yes. But my post which starts with "The previous maps had the problem that…" and the two posts thereafter are generated with a cycle hill-climber which figures out a set of best-fit cycles not limited to any size.

In this case, it would be the regions between highlighted squares which would be the break "zones" as I called them, which we would (maybe) be expecting to overlap with the 18th column, if it was removed after homophonic encipherment?

Good idea. I will look into showing the break zones on the heat map.

As Dave explained much better than me, once you replied to my initial idea it led me to agree – there are so many plausible 2-symbol cycles would this test be easier if we limited it to minimum 3 or 4-symbol cycles (i.e. FBPYFBPYFBPY, rather than FBFBFBFB)?

There are thousands of 2-symbol cycles and there are millions of 4-symbol cycles. So you are increasing rather than limiting. Within the cipher, cycles are multiplicative and you end up with allot more plausible 4-symbol cycles than you would imagine. Anyway, this problem has been fixed by using the cycle hill-climber.

There however remains a degree of uncertainty which I am pretty sure is inherent to the problem and the weaker the cycles the higher the degree of uncertainty. We are reasonable certain of the cycles in the Z408 (imagine, without knowing the solution) and quite uncertain of the cycles in the Z340.

what are 9: and 18: referring to, the change in cycle score before/after removing the 18th column? Is the problem with this test for the 340 not that, unless we perfectly fill the 18th column as it was originally we would expect a random column to do to the cycle repetitions the same as the absence of that column, therefore we would expect this result even if your 18th column was removed post encipherment?

9 is the average score for adding a 9th column (one million tests) and the same for 18. I picked 9 because it is farthest away from 18. The test may pick up whether column 18 is a better position than column 9 to add filler.

Cycle hill-climber + break zones heat maps:

Z408:

Z408 with random 18th column added:

Cycle hill-climber + break zones heat maps:

Z340:

Z340 more aggressive cycle merging:

These are very interesting observations.
However, I don’t think that there is a column left out in z340. Since the newly added column consists of completely new symbols, the period simply shifts, doesn’t it? However, this results in a misalignment. Here is an example:

In the non-expanded z340, UTP19 and Mirror+UTP15 are very similar in terms of bigrams, since they are basically the same transposition. In the z340 with an additional column (unique symbols), the same applies to UTP5 and Mirror+UTP16. In fact the bigrams are almost identical:

Mirror + UTP15 on unmodified z340

2-gram frequencies > 1:
---------------------------------------------------------
b+: 5
+4: 3
nS: 3
+c: 3
kg: 3
+B: 3
j+: 3
o7: 2
em: 2
mL: 2
MV: 2
+M: 2
Kn: 2
IT: 2
G+: 2
Ns: 2
Xu: 2
u#: 2
YA: 2
Ib: 2
BO: 2
q+: 2
+e: 2
gb: 2
tF: 2
MF: 2
8+: 2
PY: 2
zd: 2
OF: 2
TB: 2
dD: 2

3-gram frequencies > 1:
---------------------------------------------------------
Xu#: 2
q+e: 2

Mirror + UTP16 on z340 with extra column (unique symbols)

2-gram frequencies > 1:
---------------------------------------------------------
b+: 5
nS: 3
+4: 3
+c: 3
kg: 3
j+: 3
o7: 2
em: 2
mL: 2
LR: 2
G+: 2
Ns: 2
Xu: 2
u#: 2
+e: 2
YA: 2
OU: 2
BO: 2
MF: 2
8+: 2
+B: 2
zd: 2
OF: 2
TB: 2
PY: 2
dD: 2
gI: 2
IT: 2

3-gram frequencies > 1:
---------------------------------------------------------
Xu#: 2

The following image shows the relationship between UTP5 and UTP19 when a column is added:

It is interesting to note that the phenomenon also occurs when only an incomplete column is added. UTP5 can even be seen when only the last five lines are expanded:

33 bigrams at UTP5:

HERabcdVPeIfLTGgh
Nb+BjkOlDWYmnoKpq
BrstM+UZGWjqLkuHJ
SbbvdcwoVxbO++RKg
yzM+u12hI7FP+34e5
bwRdFcO-ohCeFagDj
k7+KQl8gUtXGVmuLI
jGgJp2kO+yNYu+9Lz
hnM+0+ZRgFBtrA#4K
-ucUV+dJ+ObvnFBr-
U+R571EIDYBb0TMKO
gntcRJIo7T4Mm+3BF
u#zSrk+NI7FBtj8wR
cGFNdp7g40mtV41++
rBXfos4zCEaVUZ7-+
ItmxuBKjObdmpMQGgä
RtT+Lf#Cn+FcWBIqLü
++qWCuWtPOSHT5jqbö
IFehWnv1ByYOBo-CtÄ
aMDHNbeSuZOwAIK8+Ü

Some more loose ideas. Actually, kind of a TODO or reminder for me:

– The phenomenon observed could also be the result of misalignment due to accidentally skipped symbols. It is also possible that different parts of the text were transposed differently.

– If plaintext is rotated by 90 degrees and a filler was used, it would be in the last column. Maybe this is the reason why you can see clear bigram peaks on UTP5 even when adding an incomplete column.

– Maybe it is worth investigating whether the bigram-increase on the horizontal shifted z340 (45 Bigrams, 4 Trigrams at UTP19) is related to the phenomenon shown here.

Translated with http://www.DeepL.com/Translator (free version)

It is interesting to note that the phenomenon also occurs when only an incomplete column is added. UTP5 can even be seen when only the last five lines are expanded:

That’s a +3 bigram repeat increase at period 5. I tried it with a period 5 Z408 control cipher (18th column removed) and it only increased bigram repeats by 1. Nice find.

9ZqEV%J!KpPd8He/r
=XZLGq/pMrWUFrIq
H9!_%V#JFkWBk#OeP
8R8D9c=YR8+p@+L@X
jM9=q=lABG6N9WDA
V9N)%+K(ZOeIe(TG)
P5Y6UzRFqHUU6Xkpc
8Fk+H5ZA_Pzc9d@Sp
YK(OBq!RVV_qNWW^Y
ISeI5+qMY+VWJEtT
YlLtZ^O)OeU8lPGIq
^YkGR=K7B6SEqTHr_
QIlTtS9fYN#DUAQBR
9YL_6%DdT7#5/YDL)
rztS#G_(Bd%H/@e%
9X/IF#q@MBBEXNXPHä
k9OMQ)zRUASXA^PcAü
R5ED%RM/dfk89RcRö
7lZU%LqKtZ!Eq%f=kÄ
LAq^I)P_L)N#eMWVBÜ

– If plaintext is rotated by 90 degrees and a filler was used, it would be in the last column. Maybe this is the reason why you can see clear bigram peaks on UTP5 even when adding an incomplete column.

Good reasoning, worth looking into.

– Maybe it is worth investigating whether the bigram-increase on the horizontal shifted z340 (45 Bigrams, 4 Trigrams at UTP19) is related to the phenomenon shown here.

I had the same thought but how to connect the two?

Hi Jarlve have you tried dropping the same extra column at the start and end and then comparing

if he has used skytale or p19 or any other transposition order the filler will be randomly dotted through the code. not at the end.
thats why i always liked smokies push on nulls and skips theory

Here’s another cipher that’s acting strange. This time four lines were alternately extended and four lines kept unchanged:

HERabcdVPeIfLTGghä
Nb+BjkOlDWYmnoKpqü
BrstM+UZGWjqLkuHJö
SbbvdcwoVxbO++RKgÄ
yzM+u12hI7FP+34e5
bwRdFcO-ohCeFagDj
k7+KQl8gUtXGVmuLI
jGgJp2kO+yNYu+9Lz
hnM+0+ZRgFBtrA#4K?
-ucUV+dJ+ObvnFBr-ß
U+R571EIDYBb0TMKO!
gntcRJIo7T4Mm+3BF§
u#zSrk+NI7FBtj8wR
cGFNdp7g40mtV41++
rBXfos4zCEaVUZ7-+
ItmxuBKjObdmpMQGg
RtT+Lf#Cn+FcWBIqL.
++qWCuWtPOSHT5jqb;
IFehWnv1ByYOBo-Ct:
aMDHNbeSuZOwAIK8+_

Result: bigram peak of 35 at UTP5 and P19.

Periodic: (transposition, untransposition)
---------------------------------------------------------
Period 1: 25, 25 (1.69, 1.69)
Period 2: 13, 20 (-1.44, 0.38)
Period 3: 27, 24 (2.21, 1.43)
Period 4: 21, 21 (0.64, 0.64)
Period 5: 17, 35 (-0.39, 4.31) <---
Period 6: 15, 15 (-0.92, -0.92)
Period 7: 26, 19 (1.95, 0.12)
Period 8: 17, 19 (-0.39, 0.12)
Period 9: 16, 21 (-0.65, 0.64)
Period 10: 20, 17 (0.38, -0.39)
Period 11: 20, 15 (0.38, -0.92)
Period 12: 17, 15 (-0.39, -0.92)
Period 13: 17, 19 (-0.39, 0.12)
Period 14: 18, 18 (-0.13, -0.13)
Period 15: 14, 17 (-1.18, -0.39)
Period 16: 23, 19 (1.17, 0.12)
Period 17: 19, 19 (0.12, 0.12)
Period 18: 19, 21 (0.12, 0.64)
Period 19: 25, 35 (1.69, 4.31) <---
Period 20: 31, 19 (3.26, 0.12)

I can do a test today or tomorrow by trying all combinations of line extensions. With 20 rows, this results in 2^20 combinations, so in total 1.048.576. I don’t know if this makes sense, but trying won’t hurt.

I had the same thought but how to connect the two?

I’m going fishing today, have this week off. So plenty of time to think about it. Maybe some ideas pop up.

if he has used skytale or p19 or any other transposition order the filler will be randomly dotted through the code. not at the end.
thats why i always liked smokies push on nulls and skips theory

You mean like z408 but with additional transposition? So n fill letters at the end of the plaintext and then do the transposition? This should be solved by AZDecrypt, I think. But if these fillers were left out during the transposition, you would have a real problem. In this case, you would have to know all positions before the untranspose and fill them up with symbols.

Largo:
My guess he wrote out his plaintext story on graph paper.
He had filler at the end (nonsensical junk) but only because its hard to end a sentence just perfectly to fit the 17 letter row.
Say If he had 6 places left he could have used theend or hahaha not really junk.
Next he uses a transposition technique in plaintext which also scatters the filler sporadically througout. Different transpositions give different filler positions (clusters).
Finally he uses his substitution tequnique of symbols and hopefully he did not make too many mistakes in this process.

the above is easy pen and paper stuff