– "Batch ciphers (substitution)" now supports plain text accuracy testing, a example can be found in the "Ciphers/Batch/" folder.
Example:
iterations=500000 output_sub_directory=408 9%P/Z/UB%kOR=pX=B WV+eGYF69HP@K!qYe MJY^UIk7qTtNQYD5) S(/9#BPORAU%fRlqE k^LMZJdrpFHVWe8Y @+qGD9KI)6qX85zS( RNtIYElO8qGBTQS#B Ld/P#B@XqEHMU^RRk cZKqpI)Wq!85LMr9# BPDR+j=6N(eEUHkF ZcpOVWI5+tL)l^R6H I9DR_TYrde/@XJQA P5M8RUt%L)NVEKH=G rI!Jk598LMlNA)Z(P zUpkA9#BVW+VTtOP ^=SrlfUe67DzG%%IM Nk)ScE/9%%ZfAP#BV peXqWq_F#8c+@9A9B %OT5RUc+_dYq_^SqW VZeGYKE_TYA9%#Lt_ H!FBX9zXADd7L!=q _ed##6e5PORXQF%Gc Z@JTtq_8JI+rBPQW6 VEXr9WI6qEHM)=UIk solution_plaintext= ILIKEKILLINGPEOPL EBECAUSEITISSOMUC HFUNITIAMOREFUNTH ANKILLINGWILDGAME INTHEFORRESTBECAU SEMANISTHEMOATDAN GERTUEANAMALOFALL TOKILLSOMETHINGGI VESMETHEMOATTHRIL LINGEXPERENCEITIS EVENBETTERTHANGET TINGYOURROCKSOFFW ITHAGIRLTHEBESTPA RTOFITIATHAEWHENI DIEIWILLBEREBORNI NPARADICEANDALLTH EIHAVEKILLEDWILLB ECOMEMYSLAVESIWIL LNOTGIVEYOUMYNAME BECAUSEYOUWILLTRY TOSLOIDOWNORATOPM YCOLLECTINGOFSLAV ESFORMYAFTERLIFEE BEORIETEMETHHPITI iterations=1000000 output_sub_directory=408_part1 9%P/Z/UB%kOR=pX=B WV+eGYF69HP@K!qYe MJY^UIk7qTtNQYD5) S(/9#BPORAU%fRlqE k^LMZJdrpFHVWe8Y @+qGD9KI)6qX85zS( RNtIYElO8qGBTQS#B Ld/P#B@XqEHMU^RRk solution_plaintext= ILIKEKILLINGPEOPL EBECAUSEITISSOMUC HFUNITIAMOREFUNTH ANKILLINGWILDGAME INTHEFORRESTBECAU SEMANISTHEMOATDAN GERTUEANAMALOFALL TOKILLSOMETHINGGI output_sub_directory=408_part2 cZKqpI)Wq!85LMr9# BPDR+j=6N(eEUHkF ZcpOVWI5+tL)l^R6H I9DR_TYrde/@XJQA P5M8RUt%L)NVEKH=G rI!Jk598LMlNA)Z(P zUpkA9#BVW+VTtOP ^=SrlfUe67DzG%%IM solution_plaintext= VESMETHEMOATTHRIL LINGEXPERENCEITIS EVENBETTERTHANGET TINGYOURROCKSOFFW ITHAGIRLTHEBESTPA RTOFITIATHAEWHENI DIEIWILLBEREBORNI NPARADICEANDALLTH output_sub_directory=408_part3 Nk)ScE/9%%ZfAP#BV peXqWq_F#8c+@9A9B %OT5RUc+_dYq_^SqW VZeGYKE_TYA9%#Lt_ H!FBX9zXADd7L!=q _ed##6e5PORXQF%Gc Z@JTtq_8JI+rBPQW6 VEXr9WI6qEHM)=UIk solution_plaintext= EIHAVEKILLEDWILLB ECOMEMYSLAVESIWIL LNOTGIVEYOUMYNAME BECAUSEYOUWILLTRY TOSLOIDOWNORATOPM YCOLLECTINGOFSLAV ESFORMYAFTERLIFEE BEORIETEMETHHPITI
With detailed output afterwards on how the solver and n-grams performed:
AZdecrypt batch ciphers (substitution) for: Accuracy batch.txt 5-grams_english_practicalcryptography_wortschatz.txt --------------------------------------------------------- Items processed: 300 Average score: 24019.99041 Average IOC: 0.06723 Average multiplicity: 0.27044 Processing time: 81.11 seconds - Per cipher: 0.27037 seconds Average accuracy: 66.30% ------------------------------- Accuracy equal/over 0%: 100% Accuracy equal/over 10%: 82.66% Accuracy equal/over 20%: 71.66% Accuracy equal/over 30%: 66.66% Accuracy equal/over 40%: 65.66% Accuracy equal/over 50%: 64.66% Accuracy equal/over 60%: 63.00% Accuracy equal/over 70%: 61.66% Accuracy equal/over 80%: 60.00% Accuracy equal/over 90%: 56.33% Accuracy equal/over 100%: 30%
Also, in the solution plain text one can use the asterisk character to have it not counted towards the accuracy, in case the solution is defect for example:
output_sub_directory=408_part1 9%P/Z/UB%kOR=pX=B WV+eGYF69HP@K!qYe MJY^UIk7qTtNQYD5) S(/9#BPORAU%fRlqE k^LMZJdrpFHVWe8Y @+qGD9KI)6qX85zS( RNtIYElO8qGBTQS#B Ld/P#B@XqEHMU^RRk solution_plaintext= ILIKEKILLINGPEOPL EBECAUSEITISSOMUC HFUNITI*MOREFUNTH ANKILLINGWILDGAME INTHEFORRESTBECAU SEMANISTHEMO*TDAN GER***ANAMALOFALL TOKILLSOMETHINGGI
This new functionality was suggested by beijinghouse and it has helped allot with finding improvements to the solver.
Hi Jarlve,
I’m just not sure if I asked that already: What exactly do the numbers of the transposition solver in brackets behind the transpositions say? For example:
Offset column order (311+2,X:80)
L-route(181*2,TP,C:7)
Mirror(23*15)
Occasionally I would like to perform a transposition tested by AZDecrypt myself. How can I interpret the data? Here is an example output (comes from z340, which was manipulated manually before):
Score: 21051.37 IOC: 0.07605 Multiplicity: 0.18529 N-grams: 464 PC-cycles: 951 L-route(151*3,UTP,C:6) Reverse Period(UTP,P:19) ATFIRSTARTSWHENDT INSEIDENTOIDANTOF BIRDINAMEDINTEDAS CRONESHHISNTSHERS WASHOWTOORGASESOF THEINGWANDANODALB ASROONEDIINALOWMA RGESTWILLASALSOOF THENALTRANTSTODIN AMELOATOFFISHERTE TROLONEIDENTAILSE SLAINHASLESSTORCO NNEEMORINCECENTOR TEMCITATTRALSMANB ROASSTOONESOROOND OTHEEARPRINGSACTS TFIERSCLAIMMEDTOR TESTBEATNAMSOFREE SITEOFTHROWSTLISM ICALIFIEDRECEASMH
I also have a feature request if you don’t mind: It would be great when the output of the transposition solver also contains the transposed ciphertext. Is that a lot of effort to implement in the next release?
I’m just not sure if I asked that already: What exactly do the numbers of the transposition solver in brackets behind the transpositions say? For example:
You’ve asked. Anything like 311*2, 181*2 or 23*15 are the cipher dimensions the operation is performed in.
Offset column order (311*2,X:80)
Here X is the offset, perhaps I should denote it with O instead.
L-route(181*2,TP,C:7)
I discussed the L-route transposition in the smokie’s big transposition thread. TP means that it is transposed and UTP stands for untransposed, the inverse operation. C stands for corner, in what corner the operations starts, the corners of a square are counted from left-to-right, top-to-bottom, but each corner has 2 directions so it ends up with 8 possibilities here. viewtopic.php?f=81&t=3196&start=1240#p69123
Occasionally I would like to perform a transposition tested by AZDecrypt myself. How can I interpret the data?
I agree that this is difficult at the moment, most of the operations are also in the Functions, Transposition menu but not all.
I also have a feature request if you don’t mind: It would be great when the output of the transposition solver also contains the transposed ciphertext. Is that a lot of effort to implement in the next release?
Will look into it but not sure.
Thank you very much for the explanation!
AZdecrypt 1.16 will be released within the next couple of days. Here’s a little preview:
It is a solve of the W.B.Tyler 2 cipher, first solved by Gil Broza in 2000, and first solved automatically by AZdecrypt in 2016. This solve is with beijinghouse’s new 7-grams which fit in 2 GB of memory only (without any compromises!) and can thus be ran on 8 GB systems.
Score: 17562.45 IOC: 0.07024 Multiplicity: 0.20658 Repeats: 1220 PC-cycles: 2744 Minutes: 12.73 ITWISEARLYSPRINGWARMANDSULT RIGLOWERTHEAFTERNOONTHSVERY MRBEMESSEEMEDTOSHAIETHEDELI COOOSLANGOPROSUOIVERSALNATU REAELADENTHEVARIOUSLNDSINGL EDBEHFUPESOSTHEROSEANDTHEME SSAERNETHEWOODLIKEINDTISWIL DFLOWERTHEYHLOWTYWAATEDSHEI AFRTGRANTSSFERINGTOTHEOPENW INDOWWHEOESATTHEAOVERSTLEAR DENCEINSHOOTSEOLUTINHERBLIS HINGFACEANDITSGENTLEMEIETYH ASMORELIEETHERIEATIAOFSIZEY ISDROMONCLOATHESAIRYRNSPIRA TIONOFDDREISCHANTHAACTUALRE ALITHOFEARTHTTNDERLYHEHALVE RGAMEDUBONHERASHERCLUSTEROU GRINGLETSWAREEDVEDLYAPOROUS ANDSTOITIVEREPAIRSANDWHETHE PERVESTEDTHERIDEINTRESIONOF THESUNLIGHTHESPRINGTODRAWTH ECURTSHUBUTSTAGENTLYSTAYEDH IMTONODEARCHARLESSHASOFTLIS ARDMUCHHATHERYOULDNHAVEALIT TLEMENTHENNOAINITRAL
And here’s a solve of the same cipher with beijinghouse’s new 8-grams:
"IT WAS EARLY SPRING WARM AND SULTRY…"
The cipher has a great deal of errors which makes decryption very difficult.
Score: 15136.98 IOC: 0.07011 Multiplicity: 0.20658 Repeats: 854 PC-cycles: 964 Minutes: 16.28 ITWASEARLYSPRINGWARBANDSULT RRGOOWEDTHEAFTERNOONTHSVERY BRPEMESSEEMEDTOSHARETHEDELI COOOSLANGOMROSUOIVERSALNATU REAELADENTHEVARIOUSONISINGO EDPEHFAMESOSTHEROSEANDTHEME SSAERNETHEWOODLIKEANITISWIL DFLOWERTHEYHLOWTYWAATEDSHEI AFRTGRANTSSFERINGTOTHEOPENW INDOWWHEOESATTHEAOVERSTLEAR IENCEINSHOOTSUOLUPINHERBLIS HINGFACEANDITSGENTLEBEAUTYH ASMOREOIEETHERREATIAOFSIZEY ISDROMONCLOATHESAIRYRNSPIRA TIONOFIDREASCHANTHAACTUALRE ALITHOFEARTHTTNIERLYHEHALVE RGAMEDUPONHERASHERCOASTEROU GRINGLETSWAREEIVEDLYAMOROUS ANDSPORTIVEREPURRSANIWHENHE PERVESTEDTHERIDEINTRUSIONOF THESUNLIGHTHESPRANGTODRAWTH ECURTSHUBATSTAGENTLYSTAYUIH IANDNODEARCHARLESSHASOFTORS ARDMUCHHATHERYOULDNHAVEALIT TLEMUNTHENNOAINATRAL
AZdecrypt 1.16:
Download: https://drive.google.com/open?id=1vB1G8 … zthSv8seky
What’s new?
– Version 4 of beijinghouse’s 6-grams and 7-grams included which use a special low memory no-compromise n-gram system. For example, beijinghouse’s 7-grams only take up 2 GB of ram and can thus be used on 4 or 8 GB system. His new version 4 n-grams are also much more dense and boast ridiculously good convergence rates. Version 4 of beijinghouse’s 8-grams are available as a separate 4 GB download and require a 24 GB RAM system: https://drive.google.com/open?id=12CgF1 … wZPCdFN6nD Thank you beijinghouse for all your work on this!
– Added timer to solver output so that the user knows how much time to program took to arrive at a certain solution.
– Added the option the add the PC-cycles stat the solver output "(General) Add PC-cycles to file output format".
– Fixed IOC inflation bug with Row bound solver that happened with certain ciphers.
– Fixed bug with Vigenère solver when using entropy weight other than 1.
– Fixed possible crash with Batch settings.
– Minor bug fixes and improvements all over the place.
– Compiled with the new FreeBASIC 1.07.1 release.
Awesome work! I really need to upgrade my machines to have more RAM. 
Hi Jarlve.. I have only had 10 minutes with it but it seems to be lightning fast. I am looking forward to the weekend and spending some time getting acquainted with it. I will run through some old scytales that stalled to see if I can get any better results.
Great work.
Cheers
Keep up the great work Jarlve!
Thanks guys, it means allot.
I really need to upgrade my machines to have more RAM.
RAM in computers used to double every few years but now it has stagnated – just when we need it.
Hi Jarlve.. I have only had 10 minutes with it but it seems to be lightning fast. I am looking forward to the weekend and spending some time getting acquainted with it. I will run through some old scytales that stalled to see if I can get any better results.
Cool, try it with beijinghouse’s v4 7-grams if you can! If not his 6-grams will do fine also.
Awesome work! I really need to upgrade my machines to have more RAM.
Just to clarify, my new v4 8-gram file only needs 14.35GB of memory to load.
I’ve been developing it on a system with 64GB of ram, but it’s possible you could load and run it on a 16GB ram system if you close absolutely everything else. Might be useful for testing the v4 8-grams to test how powerful they are.
I may create a lower memory version of 8-grams in the future that only needs 3.6GB of mem if people say they want it. But it’s also possible I’ll make a higher performance 8-gram file that needs closer to 52GB as well.
My advice would be to get at least 64GB of memory if you want to be able to use the best ngram files in the near future.
Out of curiosity, what is the goal of producing higher-order ngram models? Are you guys observing higher quality solves using those models? Or are there some other benefits to using them? I was theorizing that if you use ngram models of too high order, the hill climber has trouble converging. I haven’t proven that by any means, but I did some experimentation with LSTM models using Keras. You can train models of any input size, so theoretically you could run the equivalent of a 340-gram model against the 340 with a very low memory footprint, although at that high of an order it’s basically useless. I’ll do a little more research on that but hoped to gain some insight from you guys if possible.
EDIT: Here’s a link to the project where I implement the LSTM model. I hope to make this a sister project of Zenith, but it’s not ready to be released yet. Just wanted to share in case anyone has python/keras experience. https://bitbucket.org/beldenge/zenith
Out of curiosity, what is the goal of producing higher-order ngram models?
To be able to solve more difficult ciphers/hypotheses/problems of higher multiplicity.
I was theorizing that if you use ngram models of too high order, the hill climber has trouble converging.
True, it is something that me and beijinghouse have discussed also. But there are ways to overcome the convergence problem: faster hardware, denser n-grams, solver algorithm improvements and/or including lower n-gram sizes such as zkdecrypto.
In general the hill that the solver tries to climb grows more narrow/spikier as the n-gram size increases because neighbouring n-gram variations become more sparse, for example the 8-gram "DISCUSSIO" will have a good score but a 1 letter change like a "DILCULLIO" may have a value of 0 and it becomes more and more an all or nothing situation.
Thanks Jarlve, that makes a lot of sense. And when you say "denser" n-grams, can you please elaborate? Is it simply that the model is built with a larger corpus of data, so the "good" n-grams have more samples?
