is overnight usually sufficient to run azdecrypt on a cipher to see if it’s getting somewhere?
Nice work on that cipher smokie and Mr lowe, it is pretty hard actually. Also look at the PC-cycles stat, if the 340 is cyclic as we think it should really jump allot when AZdecrypt finds a partial solution.
is overnight usually sufficient to run azdecrypt on a cipher to see if it’s getting somewhere?
It depends. If the 340 would be the same as the 408 then it should find a solution nearly instantly. When there are other factors complicating the cipher then it could take a longer time depending on the multiplicity, cipher scheme and the speed of your computer.
i will work on it more smokey if you think it helps.
Not this time, but definitely some assistance next time if you are interested. I am working on something like the above for the 340, which probably won’t work, but I need to do it anyway. The message won’t necessarily carry over from one line to the next though, so it will be harder.
no problems happy to help’ just pm me when its up so i dont miss it . cheers
If you can then let it run overnight with 6-grams to be sure. I still need to speed optimize the row bound solver, it’s kinda slow. While at it I should probably add 2-grams and 7-grams as well.
I have totally rewritten the row bound solver with optimization in mind and it now runs almost as fast as the other solvers. 2-grams have been added plus 7-gram support and other improvements to the calculation method. Because of that, the 408 with rows of 3 characters that I showed earlier improved from being 55.63% accurate to 57.10%. And the Breaking Bad cipher for smokie has improved from being 74.41% accurate to 84.70%.
The cool thing is that it would now be possible to link it to the any of the transposition solvers. Especially the columnar rearrangement solver could benefit from that.
is there a thread for submitting homemade ciphers for analysis? if not, is there any interest in us laymen trying to fool you guys? i’ve let my idea run through azdecrypt for over a day without solving, but i’m concerned i may be missing some obvious setting that solves it in five minutes. i also get lost in some of the period analysis stuff so, if there is something i’m overlooking that makes it easy to decipher, i wouldn’t mind being able to go back and retool the idea to make it harder.
There is a problem with the solvers "Substitution + polyphones [auto] and [user]" in AZdecrypt 1.14. In the meanwhile go back to 1.13 to get them to work properly.
Wow, great update mr Jarvle!!!
Im honored y the quote!!
thanks !!! 
https://zodiacode1933.blogspot.com/
AZdecrypt 1.15:
https://drive.google.com/open?id=1YOBOX … z7FYJsVuVg
Again a huge update and I will illustrate some examples of the new functionality during the next days.
– beijinghouse made a special 8-gram system that is only 50% slower than 7-grams and uses only 8 GB of memory! It works very well and he can elaborate on how exactly he achieved this. Get his 8-grams here: https://drive.google.com/open?id=1oIHa6 … ISz5nLU94c You will still need about 12 GB of ram to use these.
– beijinghouse also came up with a speed optimization that aborts the n-gram score calculation when it is known that the new_score cannot be higher than the old_score. This works very well with sparse n-grams such as 7-grams or higher. For now it is only used with beijinghouse’s 8-grams.
– Added extensive cipher and plain text library, it can be found in the "Ciphers/" folder.
– Internally the score calculation is now using entropy instead of the index of coincidence. This may change the scores you are used to seeing a little bit. A small adjustment to the n-gram ini files has been made to reflect this. Old ini files still work via internal conversion.
– You may notice a slowdown of the solving speed (about 20%) due to a big improvement to most of the solvers such that for example 500,000 iterations are now worth about 1,000,000 iterations with a greater effect on ciphers that tend to get stuck.
– New function "Bias n-grams", bejinghouse asked for this, it can be found under "File, Load n-gram bias" and it tunes the n-grams in memory to some text of your choice. For example: to offer support for rare occurring n-grams such as names.
– "Batch ciphers (substitution)" now supports plain text accuracy testing, a example can be found in the "Ciphers/Batch/" folder.
– New solver "Substitution + units", it allows to solve a variety of cipher schemes such as masootz’s cipher or operations on symbols, rows or columns such as remove, expand, separate into 2nd key, replace and reverse.
– New option "(Substitution + sequential homophones) Sequential weight: 5", setting it lower will decrease the weight of the sequential homophones and setting it higher will increase it.
– New function "Normalize n-gram factor", the button can be found under "Options, Solvers" and it will normalize the n-gram factor to some internal plain text. This ensures that AZdecrypt performs optimally with your n-grams.
– Fixed major 1.14 issues with the following solvers: columnar transpostion & rearrangement, nulls and skips, and poly [auto, user].
– Removed many code redundancies in the source (merged solvers etc) and it is about 33% smaller.
– New function "Batch settings" to optimize temperature etc, suggested by beijinghouse.
– Solvers are using much less memory outside of the regular n-gram memory use.
– Vigenère solver works much better now.
beijnghouse’s new 7-grams: https://drive.google.com/open?id=1NFV-P … 35wlL47CFC
I’m astounded that you put so much effort into this and then make it available for free to anybody who wants it. Your generosity absolutely blows my mind. Thank you for this — I can’t wait to start using it!
I’m astounded that you put so much effort into this and then make it available for free to anybody who wants it. Your generosity absolutely blows my mind. Thank you for this — I can’t wait to start using it!
Thanks for your kind words DMW.
Here’s an example of the new solver "Substitution + units".
If you have a cipher in the input window and click on Solve with the units solver selected you will get a new window with some stuff in it. If someone may remember smokie’s cipher 5, it has 2 different homophonic keys by rows, odd even in its case. This is something the units solver can tackle.
I set Unit to "Row" and Mode to "Separate into 2nd key". And Key length start and stop at 10 so that any distribution will be covered (in this case the key length are the amount of rows). I also put Multiplicity weight at 1 to punish to solver for hill climbing the multiplicity:
And then clicked Start solver with 6-grams loaded for good measure:
Restart: 1 Hill climber: 1635/5000 @ 500000 Score: 19531.07 IOC: 0.06304 Multiplicity: 0.30588 Repeats: 1262 PC-cycles: 272 Unit: row(17,20) Mode: separate Key(1,3,5,7,9,11,13,15,17,19) DLIKEKILLINGPEOPL EBECAUSEITISSOMUC HFUNITISMOREFUNTH ANKILLINGWILDGAME INTHEFORRESTBECAU SEMANISTHEMOSTDAN GEROUSANIMALOFALL TOKILLSOMETHINGGI VESMETHEMOSTTHRIL LINGEXPERENCEITIS EVENBETTERTHANGET TINGYOURSOCKSOUTW ITHAGIRLTHEBESTPA RTOFITISTHATWHENI FIEDWILLBEREBORNI NPARADICEANDALLTH EDHAVEKILLEDWILLB ECOMEMYSLAVESIWIL LNOTGIVEYOUMYNAME BECAUSEYOUWILLTRJ
While it’s certainly not necessary to understand how the new 8-gram solver works to use it, he’s a few details on how I designed it. Hopefully this partial documentation helps someone who wants to make their own 8-gram files or modify my solver routines or just understand how it’s so space-efficient while also being rather fast.
* First off, the 8-grams are encoded in a way that allows them to load into two 4GB scoring tables rather than one massive 208GB sparse array. Each 8-gram is addressed by two 4-letter sub-grams for each half of the 8-gram. The two tables have separate encodings but there is a primary table that has roughly 85% of all the data and then a fail-over table that has the other 15%. If you glance at the code, you can see how I address the tables and use the failover call if there’s no value in the first table when looking up values in the main solver routines.
* For the file format, rather than distribute and load a 900+ million line raw text file, I made a custom file encoding method. Some of how it is encoded is just to have a concise bit-level encoding. But the trickier quirks are there so that the resulting bit-fields compress better than average with gzip. First, the file has four table, each with 65565 entries in 4 plain-text letter groupings. These are the encodings for all the 1) 1st 4-letter subgram aka letters 1-4 of the main table, 2) 2nd 4-letter subgram of the main table aka letters 5-8, 3) 1st 4-subgram of the backup table, and 4) 2nd 4-subgrams of the backup table. These encodings are only used internally during the n-gram loading routine. Someone making their own 8-grams would need to also decide which 8-grams they want to be able to represent, and lay them out the same way.
* The next 4 x 2 x 255 bytes in the file are raw binary indicating how many 8-grams are coming next with each score from 255 –> 1. This covers the number of entries present in both the main table and the backup table. This is important because the next part is the actual n-gram scoring section.
* The ngrams themselves are stored in what I’m referring to as a "spread format". Each sub-ngram in the first 4 tables implicitly corresponds to values 1 –> 65565 in this table. Because these values can be stored in 2^16 bits, I merely allocate 4 bytes for each value. But rather than put them sequentially, I order them with all the top bytes for all values of each score first, then all second bytes, then all third bytes, then all low order bytes. The reason for this interleaved encoding is only to assist gzip. These files compress 60% more because of the massive redundancy that this encoding format surfaces.
* These encoding methods for the 8-grams and their internal table representation are a bit extreme. But they work really well. Download time, loading time, and system memory requirements are all cut by factors of 20. And the iteration time is not much worse than 7-grams despite being run over a much larger state space.
Happy to answer other questions about the 8-gram solver if others are curious.
It is an amazing piece of work beijinghouse and I am very happy to have your 8-gram system in AZdecrypt! It certainly pushes what we can do on the 340 in terms of hypothesis testing.
If you look at the ini file of the beijinghouse’s 8-gram file you see:
N-gram size=bh8 N-gram factor=36.039 Entropy weight=1 Alphabet=ABCDEFGHIJKLMNOPQRSTUVWXYZ Temperature=600
The "bh8" at N-gram size tells AZdecrypt to branch off to beijinghouse’s n-gram system.
There’s also a 2-gram solver in 1.15 which can do a 71.32% accurate decryption of the 408:
Score: 23419.11 IOC: 0.06264 Multiplicity: 0.13235 Repeats: 646 PC-cycles: 6841 ISIKEKADSANGPEOPD EBECOURAITISSOLUC HFUNATAULOREFUNTH ANKINDINGWASSGILE ANTHEFORMERTBECOU SELONISTHALOOTMAN GERTUEINOLODOFAND TOKINDSOLETHANGGA VESLETHELOOTTHRIN DINGEXPAMENCEATAR EVENBETTERTHINGAT TINGYOURMOCKSOFFW ITHOGARSTHEBESTPO RTOFATIOTHIEWHENI MAEAWINDBEMEBORNI NPARISACAUNMOSSTH EAHAVEKISSESWINDB ECOLELYRNOVESIWID SNOTGAVEYOULYNALE BECOUSEYOUWISNTRY TORDOIMOWNOMUTOPL YCONNACTINGOFRSOV ESFORLYOFTERDIFEA BEORIETALETHHPATA
This can be further improved by using the "Substitution + sequential homophones" solver and tweaking the "Sequential weight" setting in the "Options, Solvers" menu. In the case of 2-grams I reduced the sequential weight all the way down to 0.2 (from 5) and got a 91.42% accurate solve on the 408:
Score: 16967.41 IOC: 0.06256 Multiplicity: 0.13235 Repeats: 862 PC-cycles: 12958 IDIKEKILDINGPEOPL EBECAUSEITISSOMUC HFUNITISMORAFUNTH ANKILLINGWIDSGAME INTHEFORRESTBECOU SEMANISTHEMOOTSAN GARTUEANOMALOFALL TOKILLSOMETHINGGI VESMETHEMOOTTHRIL LINGEXPERANCEITIS EVENBETTERTHANGET TINGYOURROCKSOFFW ITHOGIRDTHABESTPA RTOFITIOTHAAWHENI SIEIWILLBEREBORNI NPARASICESNSADDTH AIHAVEKIDDESWILLB ECOMEMYSLOVESIWIL DNOTGIVEYOUMYNAME BECAUSEYOUWIDLTRY TOSLOISOWNORSTOPM YCOLLECTINGOFSDAV ESFORMYOFTERLIFEE BEORIETEMETHHPITI
