KryptoK4 Lessons

#KryptosK4 #Lesson1 The goal of this handbook is to give each individual the opportunity to learn, adapt and overcome by using simple techniques used in a correct way to reach your end goal. You will notice that I will explain techniques, allow you to make mistakes, why we need to repeat, repeat and repeat. Why we need to do, not talk. When you don't do your homework, I'm not your problem. Try to get as much as possible done by yourself, you will thank yourself for it later. I usually give you 1 or more weeks to figure out, what I told before. We will start with basic things. And we gradually go towards more and more complex things. This is not a guide towards the solution in a quick way. You have to take the turns, the corners, the mistakes, the frustration, ...The solution only makes sense...When all the logic was applied. I have seen many people claiming they have the solution. You will find out, why I know they don't have them. And you will see what I mean with the east problem. So enough talking. Let's repeat what we will be working on: The difficulty of the encryption was around nine out of ten. Edward Scheidt said that his intention was for it to be solved in five to ten years. I don't provide the end result all the time, but sometimes I do (as in Lesson 1) so you can work with that. And grow for yourself. When you enter the world of cyber security you have 2 kind of people: the showoffs and the really gifted people. I can assure you, if you pursue your dreams and have the right dedication, you will make progress. These lessons, as you will notice, aren't meant for the 'quick solution showoffs'. They lack the real determination to solve complex problems. For #Lesson1 #FrequencyAnalysis you need: pencil and paper and the K4 ciphertext. The goal of this task is to count each character seperately and see how often they appear. By doing so we can start our #CryptoAnalysis. This is your end result normally:
A: 4,B: 5,C: 2,D: 3,E: 2,F: 4,G: 4,H: 2,I: 4,J: 3,K: 8,L: 4,M: 1
N: 3,O: 5,P: 3,Q: 4,R: 4,S: 6,T: 6,U: 6,V: 2,W: 5,X: 2,Y: 1,Z: 4
If you look at this part:
OBKRUOXOGHULBSOLIFBBWFLRVQQPRNGKSSOTWTQSJQSSEKZZWATJ
KLUDIAWINFBNYPVTTMZFPKWGDKZXTJCDIGKUHUAUEKCAR

I have placed the E and the H in bold. So before I let you start your homework I will give some tips. Because it is also intended for each individual who is new to cryptoanalysis. Try to figure out what the cryptographer was trying to do. Without knowing the hints. Why would you start with a specific region. What makes it that you might give this a shot. As a cryptanalist, which direction do you want to go first. In other words, how would you make sure that, when starting easy, you can make a lot of progress. Now you did get some hints. Try to use them. And try to find ways to use them with little effort. This was lesson 1. I will come back next week, with lesson 2: The Matrix Revived.

#KryptosK4 #Lesson2: The Matrix Revived. In the previous lesson people could have done several tests/examinations. For example they could have done a Kasiski examination where you look for 3 or more lettergroups that repeat. Another examination was the kappa or friedman test. There are still others, but we will come back to them in later lessons. Today you will try to understand to what looks like the main mechanism. As you will see in future lessons, we try easy to get as much as possible to gain more and more insight towards what the real implementation could be. Since you got a hint: we do what he did. So we develop several alphabets and we write them down, just underneath each letter. But we start with what we know. So for each tip we got we write underneath the letters of the different alphabets. We repeat this with eg. a 24 letter alfabet, 25 (here we remove 2 letters), 27, 28 (here we add punctuation for example)... We had 4 Q's from the first lesson. So if we know what the Q stands for, we have decrypted 1 letter completely. We only need to find the values for the other 2 Q's because we know what the first 2 need to be. We also see that we had 6 S. We got 2 for free. 5 of them lie in the direct neighbourhood of one of the tips. It also seems that the Q and the S encrypted letters are dancing with each other. It means they are strongly coupled to each other. If you find 1, you probably can figure out the other. Also try to figure out what the alphabet length could be and why. Make sure you keep your notes for later, when you get stuck. In the attachment, you will see an example of what you need to do for homework. Next week we will talk about lesson 3: Detecting hidden hints.

#KryptosK4 #Lesson3: Detecting hidden hints.

The goal of this lesson is to see if we got hidden or secret hints and if we did find them. Or did we miss hints in those 35 years?
Based on your experience you may not always detect a hidden hint. The goal of this lesson is not to start with the implementations immediately, unless you have loads of time. But it gives an insight in how you could have detected the meaning of something that at first was rather meaningless.

1) Alberti: If you look at the tabula recta on the right of Kryptos, you will notice that after the Z it continues with A,B,C and D. The Alberti cipher was a polyalphabetic substitution encryption algorithm. It was the foundation for the more complex polyalphabetic substitution encryptions used in the previous century.
2) Thritemius: The tabula recta can also refer to Thritemius. This is again a polyalphabetic substitution encryption algorithm. The tabula recta is used to transform plain text into ciphertext by shifting the rows. It works similar to Alberti’s disc.
3) Vigenère: The tabula recta got famous by another cryptographer Bellaso. This encryption is again a polyalphabetic substitution encryption algorithm. This works different compared to the other 2 but in a way it works similar. Each row shifts the alphabet 1 position like Thritemius. So you get 26 alphabets that follow a Caesar encryption.
4) Beaufort: Yet another encryption algorithm and yet again a polyalphabetic substitution encryption algorithm. It also uses a tabula recta.
5) DIANA: DIANA is again a polyalphabetic substitution encryption algorithm. DIANA uses the Beaufort tabula recta.
6) HILL: If you look at the tabula recta at the right, you see an additional L that forms HILL. This is both an encryption algorithm and a technique called Hill climbing. The hill encryption algorithm is technically a block encryption algorithm and thus a polygraphic substition algorithm. But in a broader context it is again a polyalphabetic encryption algorithm. The reason is because HILL uses a matrix to perform (complex) substitution for a full block. Hill climbing is something we discuss in a later lesson.
7) Autokey: The autokey encryption algorithm uses a tabula recta. And again this is a polyalphabetic encryption algorithm
8) One Time Pad: DIANA is an example of a One Time Pad, but you also had MEDEA and ORION (successors of DIANA).
9) ALB(E)RTI: We notice again ALBERTI, but this time in the KRYPTOS text itself. Almost at the end you will notice the letters AYCR, LB and NRTI. Start at the A, go down 1 row, you have the L. Then 1 right, you have the B. Dan 1 down and you have ALBRTI. Also instead of going 1 down. Look at EB and ORIT. You go 2 down now. You get the E then we go 1 up and you have RTI. But we can even do it with ORIT underneath. They all can form ALBERTI. Where does this start? 5 rows before the text of Kryptos K4.
10) BERLIN CLOCK: This can refer to the Enigma Uhr. An expansion for the Enigma I used by the Luftwaffe.
11) EAST NORTH EAST: This can refer to potential actions you need to do. So it can provide more details about the used implementation. We come back on this for the first time in lesson 25.

In the meantime, someone in the room still doubting we are watching a polyalphabetic encrypted text?

So your homework: try to get more information about the hints, techniques and the algorithms.

#KryptosK4 #Lesson4: You need abstraction of an implementation

It's a short lesson. So lesson 3 revealed that we are dealing with a polyalphabetic substition. Unfortunately there are many implementations of this kind of encryption. So we could be busy for a long time if we focus first on implementations. So instead we want a more general approach. If something is based on something else, then the implementation is different, but the idea remains the same. For example: Vigenère, Beaufort, Thritemius, Alberti and Variant Beaufort. They all have their specific implementation. But in general they all just use a tabula recta and thus 26 caesar alphabets. So the basic concepts for these encryptions is eg: you have an alphabet on top, an alphabet on the left and a mother alphabet that changes 1 position in the next row. This is just one example. Another one is eg. ENIGMA. When I talk about ENIGMA people always think about the 3-x rotor systems with their specific implementation. But again the main idea needs to be extracted. What does it do and why does it do that. If you look at the 'bombe' of Alan Turing or 'bomba' of Marian Rejewski they had their purpose. But in that history you also spot that the usage of that encryption was used in a way that made the whole encryption weaker. The weakness was that by encrypting the trigram twice, people could follow how it worked internally. Both bombs were in essence based on the 'grill' method. Now people have a tendency to say: it's not ENIGMA because you can't encrypt into the same letter. But if you look at the basics of ENIGMA, there are ways you also work with 3 rotors for example and can encrypt into the same letter. You basically use the fundamentals of an encryption instead of going after a known implementation. If your fundamentals are correct, you will make progress. If the fundamentals are wrong, you won't make progress and have to rethink. For example: you can rewrite a rotor system by using the fundamentals as an alberti disk with 3 alphabets. It is basically a 3-rotor system. The key difference between abstraction/generalising an idea and implementation is that in the first you gain insight. The second is just knowledge. Insight requires training. Knowledge can be teached. And so in the lessons we will write our own code. Based on insight. But if you don't know what you should be getting or how it works...Then code won't solve anything. So apart from code, it is also good to have something on paper as well. If you have 2 ways to do the same, you can verify if you are doing a good job in cracking Kryptos K4. If you have 2 ways (pen and paper and software) or even 2 types of implementation from a generalized idea (so 2 methods visually different) you can detect human errors and software coding mistakes. You will notice in the lessons that the trained eye spots what needs to be done, while an untrained eye just keeps busy brute forcing.

Next week will be about shift problems and in lesson 6 we will discuss why even without hints, we would have attacked the same region of encryption letters first.
People might think it is all going too slow, but if you train yourself each week on a certain aspect you will notice that at the end you gained insight. The only way to get it, is by doing it.

#KryptosK4 #Lesson5: Shift problems

This is about understanding problems of your shift and the connection with the alphabet length. And what this can cause in your decryption process.

In lesson 2, we followed his example. So we used his pattern and then verified how far can we decrypt using this simple decryption method. But there is also a connection between having an incorrect alphabet length and the shift. If your alphabet is 1 letter short, you need to increase the shift with 1 a bit sooner. If your alphabet consists of 1 character too much, then you need to lower your shift with 1 at a given spot. Now if you did have the correct alphabet length...And you need to shift, are you going to do it?

So let's look at the technique used in lesson 2. You should quickly have detected ARE DECODED. But in case that was too hard to spot, let's do it step by step (see attachment). Who was doing his homework correctly used several alphabets.

So the reason why you need to take both the S and the Q letter increase together is that you spot it easily. If you just took the pattern for the Q and afterwards redid this for the S then you might not have spotted this. But by doing them together we see the Q became an A at QSJQSSEKZZWAT and the next Q became a D. Now we also followed the S and the S after the first Q is a Z. And if you now look at the QSJQ section then in fact you want to be 1 shift further. They are consistent but you want them to be 1 letter further. Because then you can form ARE and the J decrypted becomes an R. And just after this section you have the SS which you want to become DE and not EF.

So today we show that if you want to have this from an implementation perspective, that's not so easy...Which implementation was used? But from an analyst perspective, this is just simple. So if you just use this simple alphabet then it's logical, but which implementation do you need to use?

Option 1: shorten the alphabet. OK, which letter will you remove? And just look at lesson 1...You have 26 decryption letters. So is it logical to drop a letter?
Option 2: we change the alphabet, since these letters need to come earlier in the sequence, we move them 1 position earlier in the alphabet
Option 3: we change nothing but we note down, we are going to do a shift +1 here on position X for letter Y.

So next week: Lesson 6: The Dune Effect. How could we have detected the same weakness without hints?

#KryptosK4 #Lesson6: The Dune Effect.
If we didn't receive any hints...How would we have spotted a weakness? Well first, let me show you the section (see first attachment).

What we notice here is that it's the same section as the part that can be easily decrypted. So if we find out what QQ stands for and we know what SS stands for, then at a given moment we also get the sequence QSJQSS. In other words, you could have used the crib technique.

If you ever watched the movie Dune or The Neverending Story then think of the sandworm (Shai-Hulud) or the dog that you were on. It will help to memorize what I'll explain. You are jumping on a rotor train. You probably wouldn't have thought of it, but what you see in the first attachment in bold are actually shift exposers. In other words, if the shift is right, you can build further on this mistake. Now I deliberately also made a mistake so you can see what happens if the K is decoded in a wrong spot. As you can see (or imagine) the full logic is becoming far more complex depending on the wrong use of an alphabet, the shift or a crib. In other words, when you use a crib, the rest must also remain logical. But in essence you will attack the same spot. To decode it, you try to use the same shift and see what you can decrypt. When you do challenges you often think of are decrypted, are decoded, are reverted, … If we apply this logic we spot 'ARE DECODED'.

So let's do it step by step (next week we come back with another aspect). The distance between the SS and the ZZ at the end is 2 letters. If we would have chosen 'areREveRTed' we would get shift problems. The R and the T don't follow up each other (with a shift of +1 and a standard alphabet). If we tried 'areDEcrYPted' then de D and the E follow each other (with a shift +1) but the Y and the P don't. Now suppose the shift changed between the SS and ZZ. Then we get yet another story. They possibly no longer use the same shift. This would give a crypto analyst a more complex situation. Not unsolvable, but more complex. So in order to figure out the possibilities you will use these 'shift exposures' to try to form words based on a different shift. Eg. shift -25, -24, …, +1, +2, +3. In other words: shift climbing (as I call it). The fact that you have 6 of these shift exposers following up each other relatively short is a crucial weakness that never should have been created in the first place. If the cryptographer paid attention, then for this challenge, he would have put more effort into making it more complex.

So next time, when you see these double letters...You as an analyst will say: whooooaaaa, let's ride the sandworm.

In the next lesson we will come back on this weakness: QSJQSS.

#KryptosK4 #Lesson7: Rodding
Rodding is a technique founded by Alfred Dillwyn Knox in 1937. If you look up Enigma, you’ll notice that the versions that didn’t use a steckerbrett had a different disc than the ones that did use it:
QWERTZUIOASDFGHJKPYXCVBNML vs ABCDEFGHIJKLMNOPQRSTUVWXYZ.

So why explain it? Because you need to get to a thorough understanding of basics principles. For example how do you break the alphabet of the right rotor if you don’t know the middle or left rotor?

The main idea of rodding is that you start with an imaginary disc. So you have the fast rotor (right rotor) which in ENIGMA had 26 contacts and a reflector system. Now if you for example pushed the N, the right rotor advances 1 position. Now you replace the rest of the system with an imaginary disc. And you notice it presents the Q. Because you have 26 positions in the standard alphabet to start with, and connect it with a rotor, you have 26x26 connections. The stable disc (your input) can consist of 26 starting positions and the rotor can have 26 starting positions. This can be translated into a tabula recta form. On the left you have the imaginary contacts of the disc. At the top you have the 26 possible starting positions of the rotor. Now you can make connections between the stable disc (input) and the imaginary disc (after it passed the rotor). Now how do you need to understand this? If your ring setting of ENIGMA was Z, in other words position 26 then in your tabula recta you would search for the imaginary result. And in this case at the right (because it was the 26th position or the Z position) you will find the letter. In this case like Thritemius. So if you know which mechanism was used and you know the encrypted letter, you can find back the original letter.

I would say, try to practice rodding as a technique. Each rotor had it’s own rodding scheme. The technique basically used a CRIB (known word) in order to decrypt more. If the rotor starting positions where wrong, you got inconsistencies. A simpler way to visualize this: it was what you did in lesson 2.

So why start about this technique? Because the distance between letters and their encrypted letters are also determined. Let’s show it with an easy example (see first attachment).

Because the provided hint by the creator was given that QQ equals NO and SS equals ST. And we said in the previous lesson we get NORTHEASTxxxxxAREDECODED. We can assume that the main mechanism is detected. The main mechanism is that it moves 1 position. In other words shift + 1 (for this specific situation). That’s of course typical for the fast rotor in an ENIGMA based system. So it basically tells you that now you can dig further and determine P, R, N, G, K and where it possibly could come in our alphabet. In other words, if you didn’t form anything yet in lesson 2. It’s now time to start creating more alphabet positions to have a better understanding what could be formed. At the moment we simplify the algorithm to it’s minimum minimorum.

But what is the additional weakness? The Q and the S arrive so short after each other that they are dancing actually. So if you didn’t know any of the provided tips you could determine through the shifting method what QQ, SS and ZZ could have been. And through QSJQSS you could also check it if made sense or not. In other words, does it form English? If you could form English words, you knew you were on the right track. So not only did this give away something about the algorithm used, but also provided a verification process so you can gain confidence on what you are doing. So this really is a crucial mistake to make.

Now to just have fast decryption without software, you could make a concentric disc with 2 alphabets. The inner disc is a disc you can turn. And you turn it after you have written down each letter. The outer disc you can’t turn. So the first time you had a N on the inner disc, you noticed the Q on the outer disc. Then you turn, now the O becomes the Q. Decryption works the other way around. Even though this is simple, you could use this mechanism as well and turn 97 positions and see how far you can get. You will notice it’s too simple of course, but try to figure out how far you can go to form English words.

Now if you didn’t form a disc, you can look at it like this. And that’s basically a tabula recta that get’s formed (see second attachment).

So to form the tabula recta, you move the Q 1 position further. For each letter you find, you do the same. Now you have 1 tabula recta or rodding form.

Who wants to go a bit further. Dilly Knox gave his team (Dilly Girls) instructions to use a specific CRIB: PERX as the first 4 characters. They would translate normally to FORX. Where X was a space. Just dig into history and look up Mavis Lever. You are basically doing something similar as the rodding technique.

For now the homework is to get as far as possible with just this simple technique. Who wants to advance further can start thinking about writing software that does this for you. And thus speed up the decryption process.

#Kryptos #Lesson8: EAST and NORTH
If you were only using a shift + 1 and 26 letters this should be your result (see attachment 1):

We know by the hints people got during the years that FLRV must become EAST, but for the R that needs to become an S we have an O at the moment... So if we look at the space between RVQQPR...That's 4 of which 2 already have a shift + 1. In other words the R that formed the T of norTh now must form an S. So we aren't going to overcomplicate stuff, let's just find out how far we get with a shift of -1. So we figure out at what position the shift needs to change in order to get the S this time. So first write out how it was originally with a shift + 1. So at the first Q, your R would translate to Q and because we were using shift + 1, the next Q the R became R. So we also get O for the R then because of this main mechanism.
If you find it hard to understand, just watch the second attachment:


But in fact you don't want that, you basically want SRQRST...We want the exact opposite, so let's try a shift of -1 for the first 26 characters.
So now you get this result (see attachment 3):

Did you remember the previous lesson? Did you look up Mavis Lever? Since you know what the F, L and V need to become, you can fill in those letters as well. You should have reached 14 out of 26 letters of the alphabet now. Maybe you can find more letters?

#KryptosK4 #Lesson9: BERLIN
So since you are gaining more and more insight, you should have noticed I also did something with the letters after position 52. I didn't do a new shift translation. So first let us do another shift + 1 (see first attachment).

We notice that this time on the spot where we want to have BERLIN we get Q?EI. If we look at the enciphered N and P we see Q and E. But in fact we want the opposite. So let's see what we get with a shift of -1 (see second attachment).

Try to find the correct shift. You have to understand what is going on, why this is happening and how you can eventually form BERLIN. This time, forming BERLIN isn't as easy as before. So how do you know you are on the right track? We will come back at this, in additional lessons.

Next week is again about BERLIN and about rotors. Today was a short lesson, next week we will talk a lot about these two things. And the next lessons will build up further on these things. And so you can gradually gain more and more insight and build up confidence. Lesson 21 will then provide something, lesson 22 is about the DIANA cipher, lesson 23 we will be cracking DIANA using Alberti. And then we have a couple of lessons about implementations and how again we can use Alberti. Lesson 28 will then be about writing your own code. And then we will be writing and changing code. Lesson 32: still need to do it. So no, even after lesson 31 you won't have the complete solution.

#KryptosK4 #Lesson10: Rotors
Let’s have a look at this result (see attachment):

What you will notice is that if you used a shift of -11 you would get C?SK?? at the spot where you wanted to have BERLIN. In other words, just like are decoded we notice another shift. If we look at the position of BERLIN, then it starts at position 64. Now if you look at the history of the ENIGMA machine, you will notice it used 3 or more rotors. It was also developed by the lesser known Dutch Marine officers Theo van Hengel and Rudolf Spengler back in 1915. During the second world war the Polish mathematician Marian Adam Rejewski broke this algorithm. Afterwards Bletchley Park used the information of Rejewski. Breaking ENIGMA was only possible by circumstances like using unsafe procedures during encryption, find codebooks and errors/mistakes done by operators. Rotor systems where heavily used in the previous century and it even was used by NATO. These rotors had a notch, so that other rotors could turn. This mechanism made it possible to have a polyalphabetic substitution encryption where each rotor on itself was a substitution encryption. One of the mistakes they made was believing that adding more rotors, made the encryption more secure. But that isn’t actually the case. Let’s look into the details. If you would start the right rotor (or the fast rotor) in a setting that it starts just after the notch, you have 25 positions that encrypt in the same manner. And then you will get random characters. In other words you can first crack 25 characters And then a new substitution is used because the middle rotor is turned. Now here we can have another 25 positions that encrypt somewhat different, before it changes again. It can be sooner too. And then the left rotor is turned. Again another 25 positions that encrypt in the same way possibly, or sooner. So if we started our right rotor right after the notch and the other 2 rotors are not just before the notch, then we most of the time the same encryption pattern: 25, 50, 75, 100, … But suppose your right rotor changed after 10 positions and again the other 2 aren’t yet near a notch then you have eg. a change after position 10, 35, 60, 85, … Now if those rotors also have a weak setting so that not only does it change very little, but also uses a weak substitution, then you can abuse another weakness. And just as with EAST NORTHEAST we noticed that there is only a minimal change required, after we did find the correct shift.

Also there is a special relation between NORTHEAST and BERLINCLOCK. If you look carefully you see that the P, the N, the K and the S enciphered letters are used in NORTHEAST. And just like the S was used in ARE DECODED, the P, the N and the K are used in BERLIN CLOCK. And there is even another bonus. The P and the N used in NORTH. Is reused in BERLIN! Not only did they use the P and the N, they also kept the distance between them the same! Let’s see this in action. The P was an R in noRth and that same P needs to become an R in beRlin, exactly 38 positions further! Now one can say, so? If 3 rotors were used, then with a distance of 38 and at least 60 characters we know that the third rotor must have been activated. Now we also know something else because of this. If rotor 2 and rotor 3 moved at the exact same moment, then you would get a certain shift. Suppose the third rotor is responsible for an overall shift of -10 and rotor 2 of -2 then the new shift is -12. Now if you would use a shift of -12 then things don’t add up. In other words, if 3 rotors were used, then they shifted separately! Meaning they are accountable for 2 different shifts. Eg. 1 of -1 and then afterwards 1 of -11. You now start to see that having C?SK?? isn’t as bad as you thought it was at all! So the total shift is -11 but somewhere we also have a shift of 1 making sure C?SK?? becomes B?RJ??. Whoops, did we miss something? So did he use ENIGMA and rotors? He definitely used something that works similar.

So again a mistake was made and we could figure out and understand better how this mechanism works. Let’s imagine a possible scenario: Rotor 2 starts with a letter that causes a shift of -1, you can rewrite this as +25 (in a 26 letter alphabet). It’s exactly the same. Now we move on and then the shift becomes + 1. The third rotor could be responsible for a big shift (eg. 10), whilst the second rotor is responsible for a small shift (eg. 2). So suppose that rotor 2 is causing a shift of +2 or -2 and rotor 3 is adding an additional shift of +9 or -9, we get a total shift of +11 or -11.

We are getting confident that we are looking at a (quasi-)nonperiodic polyalphabetic substition. You see we get more and more details and more and more confidence on what we are looking at. Enigma happens to have a non-periodic mechanism. Do we know others? Sure: Alberti, One Time Pad and Running Key Cipher. And as you can see now, the plot thickens. Maybe we can use the tabula recta, maybe we can use rotors too.

Now when playing with those alphabets...Suppose you inverted the alphabet. Can you reproduce it? In fact you can. Now your shifts need to be +1, -1 and then +15. I hope you understand why. Else have a play with it until you understand the why.

Coming back on the N and P again . You should also notice a connection between both shifts. If you did what was asked in lesson 2, you would have noticed that the letters to form BERLIN where a bit too far. But both the N and the P work as a connection. So if you ended up 2 too far for the N you also would have spotted the connection with the P. In other words, suppose you used a certain shift and you noticed that for the B of Berlin you got eg. F. Then the R of Berlin has the same issue. If you ended up 2 too far, the R will end up 2 too far as well, as long as you don’t change the shift after the B of Berlin.

But we also mentioned the S. Why? Because the S of EAST translates to itself. Just like the K of CLOCK…

So bottom line, we aren’t going to use ENIGMA as implementation, but we are aware that the mechanism uses shifts.

So to better understand the soul of Kryptos K4, you need to decrypt more letters. Now in order to not spoil everything, you will notice I won’t provide too much details in the coming lessons.