Jarlve, thanks for doing this. Your writeups are very useful and informative. I will work on linking to these results via my wiki.

Thank you doranchak, it is much appreciated.

and to inspire others to do same

Well, you inspire me to keep going and not give up on this thing.

Great job on your work so far! I still haven’t tried your solver but I promise myself to do it soon.

Well, you inspire me to keep going and not give up on this thing.

Great job on your work so far! I still haven’t tried your solver but I promise myself to do it soon.

That’s nice to hear!

Let me know If you have any issues with the program and I’ll see what I can do.

Great work.

Thanks zydeco,

AZdecrypt095 is finished and available for download AZdecrypt095. I have not yet updated the documentation.

The solver has moved up to using 5-grams from practical cryptography. Thanks to daikon for making me aware of that.

The program has lost about half of it’s speed but it’s a much more powerful solver now. There seem to be some things that it excels at, for instance, reasonable short strings of ciphertext with not too many symbols. An example of this is given by a solve of the first 8 rows of the 408 (multiplicity 0.375). Which was achieved by changing the Ioc target weight to 2 and many attempts. It now supports up to 200 unique symbols and ciphertexts up to 10.000 characters.

ilikekillingpeopl
ebecauseitissomuc
hfunitismorefurth
ankillingwillgame
intheforcestbecau
semaristhemoathan
gertueanamalofall
tokillsomethinggi

The program works great out of the box but a better solve can be made on some ciphers by changing the Ioc target weight to 2 or higher. A good example of such a cipher is daikon’s 3rd cipher (included with the program).

Have fun with it and if there’s anything, let me know.

Edit: broken links are fixed now.

Jarlve, would you be willing to share AZdecrypt’s source code with me? I am a web developer and am very interested in porting it to a web application. Ideally, I would host the web app somewhere, and anyone could go to the web site and submit cryptograms for it to solve.

Great idea doranchak!

I added 2 new tests, row flips in a 17 by 20 and 20 by 17 grid. I also added some of my other work on this forum to the My other work section. I’m now starting work on 3 new tests, exploring the possibility of the cipher having a number of rows, columns or symbols as filler up to a given multiplicity limit. Quite an exciting prospect with my new solver!

Download AZdecrypt096.

I wasn’t planning on releasing another version so soon but I needed some additional features for some of my upcoming tests.

– 10% to 15% faster.
– Removed alphabet support.
– About double the recovery rate for most ciphers.
– Fixed in and/or deflation of scores within the same solution.
– Pressing the "x" button to stop the program working is much more responsive.
– Changed the system "keys per ciphers" to "iterations per cipher", one iteration is about 300.000 keys.
– Changed the output file format to include multiplicity. Example: (15637_654_132_1.txt = score_ioc_multiplicity_number.txt).
– Normalization by multiplicity (not perfect, but will do for the multiplicty range the program is able to handle). There was too much score inflation for unsolved ciphers with high multiplicity.

When moving the AZdecrypt solving subroutine to another program I noticed that it suddenly worked much better. I have identified the problem (it made each cipher slightly polyalphabetic!) and I’m working on yet another update that will be really worth it. It’ll include another option "keys per iteration/random restart" that will be beneficial for solving hard ciphers.

Maybe it’s good that my solver was plagued by this problem since the start, because it really made me go all the way to improve upon it. In terms of the validity of the tests I did, they should still be valid since it was already solving higher multiplicity ciphers than the 340. But I may opt to rerun some tests starting from next year.

The jroberson 405 cipher still does not solve because the solution scores lower than the local optima, it will require higher order n-grams to solve this cipher. I have tested that the solver can now solve ciphers in the 0.3-0.4 multiplicity range, of course depending on how difficult the cipher is, which are another range of factors, often unknown.

That’s great news, Jarvle. I look forward to the update!

Download AZdecrypt097.

– A bug was fixed that greatly enhances the general performance of the solver.
– A new setting is introduced "keys per iteration" which allows the user to increase the search depth of the hill climber. Very benefical for hard to solve, high multiplicity ciphers.

Under ideal circumstances (high scoring plaintext) the solver should be able to operate up to a multiplicity of about 0.4.

I’m working on 6-gram support for AZdecrypt. I have used the Usenet corpus (thanks to daikon for making me aware of it). After cleaning up the corpus it still sized 23.2gb and the total size of the 6-grams is almost 500mb! Anyone happen to know an even bigger corpus?

The 5-gram solver will be kept as it is and the 6-gram solver will be introduced as a new solving module and will probably be much slower. To use the 6-gram solver your computer will probably need 2 or 4gb of RAM to work efficiently. I can’t give an ETA because I’m not sure how long the optimization process will take but it should be ready within a month or so. Also plan to improve the 5-gram solver a bit after the 6-gram release.

Jarlve-

Do you think that using 6-grams will be of benefit? I’m not saying they won’t but there is a point of diminishing returns somewhere. I have not done the testing myself, but I’ve read up on it a lot, and many researchers believe (maybe a consensus) that 3 and 4-grams are optimal.

Longer n-grams are not going to be a panacea. At some point, it becomes less about general English language stats and more about the specific words / phrases used in the corpora.

I’m not saying it is wrong to try it, but I don’t think longer and longer n-grams are going to be much help. Just my opinion.

-glurk