It is really a much larger square area than the picture, and the global maximum hill is, although the largest of all hills, proportionately much smaller than in the picture, and there are a lot of smaller little hills in the square (I know it really isn’t a square) area also. The program allows itself to step down from the smaller hills by accepting lower scores sometimes.
I am still testing my idea of allowing only 1 million iterations per division on the 5/3, and it is looking like more restarts dropping the wanderer on the square area in more places and not allowing him to wander for quite as long may be better than dropping the wanderer in fewer places and allowing him to wander for longer. That will take several more days at least.
I would like to revisit the orbiting idea and test it maybe after the idea that I am working on now. The 2.6% I said above was with nulls and skips in the exact same position, not orbiting by 1 or 2 positions or more, of which there would be more and cause small hills. I wouldn’t want to test it with a 5/5 because the area is so large, and I don’t have a grasp of how many restarts at how many iterations results in how many solves. But, maybe a 3/3, with 3 nulls evenly spread out in the top half of the message, and 3 skips evenly spread out in the bottom half of the message, with two to four rows in the middle unaffected. If we do that then we may be able to more reliably figure out how many smaller hills there are created by orbiting at 1 to 2 positions or more. Possibly no programming fix at all, but a comparison with 6/0 and 0/6. And then we wouldn’t have to wait several days before moving on to something else.
What do you say about a 6/0, 3/3 and 0/6 with fewer restarts and iterations, but still plenty enough?
Sorry if I repeat myself a lot.
Here is one with 10,000 iterations for the 5/3. Top chart orbiting by 0 positions ( same exact position ), second chart orbiting by 0-1 position, third chart orbiting by 0-2 positions, forth chart orbiting by 0-3 positions.
It is really a much larger square area than the picture, and the global maximum hill is, although the largest of all hills, proportionately much smaller than in the picture, and there are a lot of smaller little hills in the square (I know it really isn’t a square) area also. The program allows itself to step down from the smaller hills by accepting lower scores sometimes.
Indeed.
What do you say about a 6/0, 3/3 and 0/6 with fewer restarts and iterations, but still plenty enough?
We can do it but I am still working on other tests and that could take quite a while still. My workstation is running 24/7 on this.
Yes, my computer is going at it all of the time as well. Thanks.
One of the tests on smokie 5/5 just finished with 1,000 restarts each.
The test was to trade hill climber iterations for substitution iterations:
80,000 hc its @ 500,000 sub its: 1.3% solve rate
40,000 hc its @ 1,000,000 sub its: 1.9% solve rate <—
20,000 hc its @ 2,000,000 sub its: 1.5% solve rate
This shows that my original assumption to only increase hill climber iterations for increasingly harder nulls & skips ciphers is not correct. I will try to come up with a new table, something like this:
Nulls & skips: hill climber iterations @ substitution iterations
1: 1,000 @ 500,000
2: 1,500 @ 550,000
3: 2,250 @ 605,000
4: 3,375 @ 665,500
5: 5,062 @ 732,050
6: 7,593 @ 805,255
7: 11,390 @ 885,780
8: 17,085 @ 974,358
9: 25,628 @ 1,071,794 (switch to 6-grams from here?)
10: 38,443 @ 1,178,973
11: 57,665 @ 1,296,871
12: 86,497 @ 1,426,558
So the idea is to start with a number (1,000) for the hill climber iterations at 1 null & skips. And for every increase of 1 null & skip apply a factor 1.5. And the same for the substitution iterations but with a factor of 1.1. This would save the hassle of trying to figure out optimums which is nearly impossible to determine on only a few ciphers.
What do you think smokie?
You are saying that we need to optimize the ratio of HC iterations to GN iterations? Looks good. We need a unit of measurement, but it should be HC-GN ( multiplied ) instead of HC / GN ( divided ), not sure, and the table will show equivalencies as far as efficiency is concerned? Any two rows on the table should be roughly equivalent in time needed to process, is that why 1.1 and 1.5? I would round to the nearest hundred because this is an imprecise science. Maybe work on the general concept with a slightly easier cipher and move up from there.
My testing will be done in a few more days, but it seems to be shaping up that the ratio of HC iterations to restarts can also be optimized, and there is a point of diminishing return. I am processing 3 million iterations in batches with different ratios of HC to restarts. I imagine it like the big square area, and dropping a blindfolded person onto the area in different locations and letting him wander for a certain number of steps, mostly uphill but sometimes down. When he is done with that number of steps, drop him somewhere else. It is looking like there are a lot of different ratios of HC to restarts that result in basically the same number of solves.
Jarlve 5/3 With 3 million iterations so far, but I am filling in more steps:
10,000 HC * 300 restarts = 17 solves, or 176,000 HC iterations per solve
20,000 HC * 150 restarts = 22 solves, or 136,000 HC iterations per solve
30,000 HC * 100 restarts = 12 solves, or 250,000 HC iterations per solve
Better to drop the blindfolded person in 150 places and let him walk 20,000 steps than to drop him in 100 places and let him walk 30,000 steps. We could find the category with the least number of HC iterations per solve, and then, from there, maybe increase the number of GN substitution iterations?
I still have unfinished business. Some 1 million HC results:
5,000 HC * 200 restarts = 4 solves
50,000 HC * 20 restarts = 5 solves
100,000 HC * 10 restarts = 5 solves
Suggesting that you can drop the blindfolded person in 200 places and let him only take 5,000 steps, 20 places and let him walk 50,000 steps, or only 10 places and let him walk 100,000 steps and he will find the top of the highest hill the same number of times! EDIT: Note that if I multiply by 3, we will have numbers roughly the same as 10,000 * 300 and 30,000 * 100. A bell curve seems to be taking shape.
Question for you: How much larger is the search space for a 5/5 versus a 5/3? 117,000 times larger?
Any two rows on the table should be roughly equivalent in time needed to process
No, the table is progressive. I was saying that we also may need to increase the substitution iterations a bit with each increase of nulls & skips.
My testing will be done in a few more days, but it seems to be shaping up that the ratio of HC iterations to restarts can also be optimized, and there is a point of diminishing return. I am processing 3 million iterations in batches with different ratios of HC to restarts. I imagine it like the big square area, and dropping a blindfolded person onto the area in different locations and letting him wander for a certain number of steps, mostly uphill but sometimes down. When he is done with that number of steps, drop him somewhere else. It is looking like there are a lot of different ratios of HC to restarts that result in basically the same number of solves.
I was wondering what you were doing in the spreadsheet but understand now. It is a good idea and I may follow up on it.
Suggesting that you can drop the blindfolded person in 200 places and let him only take 5,000 steps, 20 places and let him walk 50,000 steps, or only 10 places and let him walk 100,000 steps and he will find the top of the highest hill the same number of times! EDIT: Note that if I multiply by 3, we will have numbers roughly the same as 10,000 * 300 and 30,000 * 100. A bell curve seems to be taking shape.
This is really good work smokie and I totally get it. I would love to see the bell curve when you are done with the tests!
smokie, this new build should no longer be able to stack two nulls at the same position, or a null and a skip at the same position. Two skips at the same position is allowed. The hill climber will now do exactly the specified number of iterations. Can you please check with your analysis tool that these things are the case? I may have missed something. Thank you very much!
https://drive.google.com/open?id=1y1D-6 … pGnhoV2euz
Another thing. Just finished a 2 day bug hunt. I noticed a MIPS slowdown of up to more than 50% when using many threads (8+) with the latest build (06162018) and eventually tracked it down to multi-threaded string operations of all things… This happened one time before with FreeBASIC’s rnd function which does not like multi-threading too. FreeBASIC issue. This has long term implications as now I have to consider minimizing threaded string operations and doing that affects the order in which AZdecrypt does things.
I fixed that problem for the nulls & skips hill climber though there remains one issue. The nulls & skips keys in the individual solve files (20429_751_188.txt etc…) are probably not the correct keys. Though what is shown in the Output window and hcinfo.txt info files is correct so it should not affect your analysis.
I will have an answer to your new version in about 7 hours at my lunch break.
I just ran Jarlve 5/3 at 10k HC and 500k Sub three restarts and will check them for orbiting on the spreadsheet at lunch, however, I did already get one solve, and it did not register on the development information which still says 0 over 22,500. Just letting you know.
Here is the solve. Only a few positions where skips are in the same place ( lower graph ).
I just ran Jarlve 5/3 at 10k HC and 500k Sub three restarts and will check them for orbiting on the spreadsheet at lunch, however, I did already get one solve, and it did not register on the development information which still says 0 over 22,500. Just letting you know.
Yes, I see. The last restart does not register its solve. Thanks for spotting that.
I just ran Jarlve 5/3 at 10k HC and 500k Sub three restarts and will check them for orbiting on the spreadsheet at lunch, however, I did already get one solve, and it did not register on the development information which still says 0 over 22,500. Just letting you know.
Yes, I see. The last restart does not register its solve. Thanks for spotting that.
Fixed and updated the 06252018 build with: https://drive.google.com/open?id=1s85vJ … IbSc-mhdYn
Here is the solve. Only a few positions where skips are in the same place ( lower graph ).
Two skips in the same place should be allowed since that is a possibility. Can you find any null and skip in the same place or two nulls in the same place?
The latest build just stopped working in the middle of a run. Need some time to sort things out properly.
