Intelligence at Midway:

The Battle within the Battle and the Men Behind the Miracle

                 December 7, 1941, a day of infamy, as Roosevelt would forever label it in his speech before Congress the following day, had brought the United States into war.  The Pacific fleet at Pearl Harbor had suffered a nearly fatal blow when the Japanese attacked without warning.  A crippled U.S. Navy began the process of patching together task forces to push back against an aggressive Japanese fleet that would string together a series of victories in the Pacific that stretched their control over the area in almost every direction.  The stunning intelligence failure leading up to Pearl Harbor would haunt the U.S. in the following months as they struggled to understand what happened and why.  Listening posts throughout the Pacific combed through their records to see what was missed, what clues had been lost, and what errors had cost them so dearly.  The intelligence station located at Pearl Harbor itself suffered the greatest humiliation since the island where it was housed had been the target.  The men who worked there, and the intelligence they uncovered would soon exorcise many of the demons of December 7 and turn the tide of war.  Their skills, personalities, and internal battles would all play a part in the story of the Battle of Midway.

“The man with the blue eyes will want to know your opinion of it.”  The blue-eyed man was Admiral Chester Nimitz, Commander-in-Chief of the Pacific Fleet.  The “it” in question was a Japanese message Commander Joseph Rochefort, chief of the Combat Intelligence Office at Pearl Harbor, Hawaii, had just decrypted.  Rochefort was presenting the deciphered message to his boss at CIO (also known as Station Hypo) Lt. Commander Edwin T. Layton.  What Rochefort knew, and what Layton was about to learn, would alter the course of the Second World War in the Pacific theatre. On May 14^th, Rochefort had deciphered the words “koryaku butai” (which meant “invasion force”) and the letters “AF” (which referred, in the opinion of Rochefort and his team at Hypo, to the island of Midway).  Layton now had the task of convincing Nimitz that what they had uncovered was a Japanese plan to launch an invasion of Midway in the central Pacific.[i]

Bringing Nimitz on board was only the first step of many.  With the wounds of the intelligence failure before the December 7^th attack on Pearl Harbor six months before still fresh, the Navy’s intelligence department based in Washington, D.C. (also know as OP-20-G) was desperate to gain bureaucratic control over operations in the Pacific.  Claiming they had authority over intelligence assessments, they set about limiting what information was analysed outside their office in an effort to consolidate operations under their own rubric.  Yet Layton believed the failure at Pearl came from just such limits placed on Pacific fleet intelligence analysts who had access only to the Japanese military message traffic, while the Japanese were sending key messages hidden within diplomatic traffic.  It seemed little had changed since Pearl.[ii]

The battle between Hypo and OP-20-G had already reached a fever pitch when Rochefort told Layton of his May 14^th decrypt.  Layton believed the Washington office analysts were far too distant from the actual arena of the war in the Pacific.  As such, they were filtering their assessments through the lens of Japanese strategic capability.  Layton’s team at Hypo, operating in the Pacific, were closer to the action and more attuned to the reality of Japanese tendencies.  To Layton, “the question of whether or not radio intelligence evaluations were to be made in DC or Pearl” was key to any victory over the Japanese in the Pacific theatre.  To swing the intelligence pendulum back to Hypo, Layton needed to convince Nimitz, who would then need to convince the rest of the Pacific brass.  Hypo needed to prove their case.[iii]

The code-breaking efforts of the analysts at Hypo had met with successes and failures alike.  The Allies had broken the JN-25 cipher system of the Japanese.  Though the code was routinely changed, the analysts at Hypo were fairly adept at breaking the revisions at a quick pace.  The code was a series of 5-digit number groups that also included “additive recipherment” to protect from garbled messages.  Though codebooks were produced as soon as the code variants were broken by Hypo, the additives involved a sequence of 18,000 random 5-digit groups.  To break such a code required the decryption team access to as many of the encrypted messages as possible.  Yet Hypo received only 60% of the total traffic, of which only 40% could be broken.  JN-25 carried roughly 90% of all the Japanese naval traffic. The enormous effort to crack JN-25 was made even harder due to OP-20-G’s reluctance to share messages with Hypo.  The limited traffic made it “like trying to assemble a jigsaw puzzle with lots of empty pieces.”  Thus Rochefort and Layton had to piece together, for Nimitz and for their detractors in Washington, the puzzle of “koryaku butai” and “AF” and show it meant an imminent attack by the Japanese fleet on Midway.[iv]

 Overcoming the scepticism of OP-20-G was a difficult task and Rochefort worked tirelessly to achieve it.  Deep in the basement of Station Hypo Rochefort poured over intercepts.  His windowless office had only one entrance which had an armed guard posted around the clock.  In “the dungeon”, as it was called, stacks of punch cards with every 5-digit JN-25 sequence individually entered on them awaited Rochefort’s analysis.  An IBM card reader helped Rochefort sort through the messages for duplicate entries, plain language phrases, and any errors or repetitive phrases that might provide a key to what the numbers meant.  So adept were Rochefort and the other Hypo analysts, they would later brag that they were able to differentiate between Japanese radio transmission technicians due to the way they struck the codes into the telegraph.  Captain Gilven M. Slonim recalled of Rochefort, “Joe’s real genius [was] assembling all these fragments and making educated guesses at the rest.” Years later, Layton remarked on Rochefort’s tireless efforts, “I could not have wished for a better and more understanding colleague in a difficult situation”.  Rochefort had “been present at the birth of the navy’s radio intelligence effort” and now his experience would be tested at a most desperate hour.[v]

After their surprise attack on Pearl Harbor, the Japanese fleet had won one victory after another in the Pacific.  In the six intervening months, they had effectively stretched their defensive perimeter thousands of miles east toward Hawaii.  They had nearly destroyed the U.S. Pacific fleet and were poised to strike another crippling blow.  The plan, launched by Admiral Yamamoto, the architect of the attack on Pearl, was to feint toward the northern Pacific, drawing the remaining American naval presence into the open, then spring the trap and deliver a fatal strike, thus clearing the way to invade Midway as a staging ground for a later invasion of Hawaii.[vi]

What Yamamoto did not know, nor would even entertain, was that Rochefort and Hypo had broken JN-25.  This marked perhaps the initial failure of the Japanese in Midway.  Their successes had bred an arrogance that blinded them to the possibility that their plans would go smoothly.  During war mapping sessions they had failed to account for their own potential losses.  Instead they were “carried through…with a sunny lack of realism…[in which] no situation could exist [where] the Japanese would not be in complete control.  [Nothing was allowed] to happen which would seriously inconvenience the smooth development of the war games to their predestined conclusion”.  The Japanese plan relied so heavily on the “strategic habit” of surprise hat no contingency was made should the Americans discover the plan in advance.  Such attitudes made it impossible for Yamamoto to permit the idea his surprise would not work.[vii]

Meanwhile Rochefort was furiously analysing as many as a thousand intercepts a day in almost absolute secrecy.  In fact so secret was the success of the JN-25 break that high-ranking officers in the Naval Department believed the Midway intelligence must have come from a spy deep within the Tokyo government. While members of OP-20-G knew about the break they were still reluctant to believe in Rochefort’s analysis.  The men in charge in Washington- Captain Joseph Redmen, Deputy Director of Communications for OP-20-G, his brother Commander John Redmen, also a deputy- were still busy asserting their authority over all communication intelligence.  They attempted to undermine the reliability of Hypo with the commanders in the Pacific, offering different assessments of Rochefort’s intercepts, and waged a personal campaign against the administration at Hypo.[viii] 

While Rochefort and Layton believed they would ultimately be proved right, the “clash of jurisdiction and personalities” was destroying the “personal loyalties and relationships that were essential to the creative effort of code breaking”.  Such actions proved to “exacerbate the inevitable differences in intelligence estimates that emerged with Rochefort at Hypo”.  As Layton would recall decades after the ferociousness of the battle of Midway, “The fight over who had made the right intelligence evaluations was every bit as fierce.”[ix]

Unable to convince Washington of their analysis of the most contentious piece of evidence, the actual location of “AF”, Rochefort devised perhaps the most ingenious, and single biggest deception of the war outside of the Normandy invasion.  His plan would use the tool he knew best- radio intelligence- to prove the target of the Japanese attack.  Hypo arranged for a message, broadcast from Midway’s fresh water processing plant, that an explosion had crippled their ability to supply fresh water to the island.  The distress call was sent using a cipher system the U.S. knew the Japanese had broken.  When Japanese air units headed to Midway for the invasion were heard radioing back to headquarters asking for emergency water supplies for the invasion forces, Rochefort and Layton had proved the attack location was indeed Midway.[x] 

Beyond the breakthrough of confirming the target, the water shortage ruse had a side effect that would aid Nimitz in his counterattack plan.  First, it narrowed the timeline of the attack since the requests for fresh water supply gave a window for the transports to land with the water.  Second, the intercepted messages also revealed the position of the Japanese convoys at a specific time allowing Nimitz to pinpoint certain targets and eventually sink much of the Japanese merchant vessels in the area.  Firmly convinced in Rochefort’s analysis, Nimitz sent Halsey a message to “return to Pearl…it appears the Jap plan another offensive in the near future…maybe in the central Pacific…designated “AF”.[xi]

In early 1942 “the attention given to Japanese submarine operations by Combat Intelligence [Station Hypo] and diligent study of their behaviour…paid off generously”.  Rochefort’s close monitoring of radio traffic provided information even when messages could not be deciphered.  The volume and concentration of the Japanese messages gave Hypo valuable clues to the location and size of the Japanese fleet.  The message traffic between submarines and seaplanes gave Hypo early clues to how such communications represented force deployment. These hints would be invaluable in the code-breaking operations for Midway and subsequently locating the Japanese fleet. “In the enigmatic world of traffic analysis and breaking ciphers, any precise information that also offers an opportunity to practice and experiment repeatedly with known quantities could help reduce the variables while trying to crack secret messages.”  Therefore the pinpointing of the Japanese attack force via the intelligence breakthroughs of Hypo allowed Nimitz to attack the Japanese carriers with the bulk of their planes still on the deck.[xii]

Before this could happen the U.S. Navy needed yet another bit of deception to spring the trap on the Japanese.  Having cracked the JN-25, and tricking Yamamoto’s forces into revealing a confirmation of their target, it was vital to keep the Japanese from learning their plan was in the open.  To convince their opponents they had fallen for the Aleutian feint, Nimitz ordered Task Force 16 to be spotted heading north toward the Phosphate Islands.  This maneuver kept the extent of the broken JN-25 code from discovery.  Had the U.S. not convinced the Japanese fleet they were steaming for the Aleutians, the Japanese may have learned their plans had been discovered and logic would lead them to conclude their codes had been exposed.  When Hypo deciphered messages reporting the Japanese had seen the task force, they knew everything was falling into place.[xiii]

Radio deception also helped convince the Japanese that Midway was vulnerable.  The U.S. knew the Japanese had broken one of its own communication systems the Navy used in the Pacific. The seaplane tender Tangier and the cruiser Salt Lake City, patrolling New Hebrides and the Coral Sea respectively, sent radio dispatches as if they were two of Admiral Halsey’s aircraft carriers in an effort to further convince the Japanese fleet that a significant amount of carrier resistance was thousands of miles away in the South Pacific.  This ruse created an even greater sense of security for the success of the Japanese attack on Midway.[xiv]

Using radio intelligence to near perfection, Rochefort and Station Hypo not only discovered the Japanese plan, they also laid the groundwork for a counterattack. Additionally they deceived the Japanese fleet into believing their own plan was working beautifully.  Even with the great intelligence work in advance, the Pacific fleet still needed several instances of luck to keep the Japanese at bay.  The attack at Pearl Harbor had failed to find all the Pacific stationed carriers in port, allowing the U.S. Navy to escape complete destruction.  After capturing a British vessel, the Germans had actually discovered information in advance of Midway that JN-25 had been broken, however they failed to deliver the documents they had captured to the Japanese in time to serve as a warning.  Additionally a recent battle in the Coral Sea that had also used Station Hypo decrypted intelligence, significantly wounded several Japanese carrier groups that were scheduled for deployment in the Midway attack.  All of these events helped boost the U.S. chances.  The odds were long, but the radio intelligence and decryptions had given the U.S. an opening.  As Nimitz and the Pacific fleet waited to pounce they could only hope their luck would hold.  Rochefort and Layton hoped their work would not be in vain.  They would not be disappointed.

The sum total of the eventual U.S. victory in Midway was stunning.  The initial force Yamamoto planned to send to Midway was overwhelming- 700 planes and 200 ships.  In materiel, the U.S. fleet was completely outmatched.  Yet the differential in damages after the battle was waged tells a different story.  The Japanese lost twice the number of airplanes (332 to 147), four times the number of carriers (4 to 1), and over eighteen times the number of casualties (2,500 to 307).  These losses significantly altered the balance of power in the Pacific- especially the aircraft carriers which were key to the Japanese success to that point.[xv]

Beyond the physical damages the psychological damage was almost as significant.  Midway was more than just a victory for the Americans, it was vindication.  While not erasing the memory of Pearl Harbor, Midway lifted the collective spirit of the Pacific fleet, which until that point had been on the ropes since the beginning of the war.  The momentum shift and morale boost that such a decisive victory brought to the U.S. was as important to the rest of the Pacific campaign as any single event, and not only changed the course of the war in the Pacific, but Europe as well.  Success at Midway allowed the Roosevelt administration to place renewed emphasis on the European theatre.  Had the Japanese succeeded in controlling Midway, Roosevelt would have had to turn almost complete attention to the Pacific.  Public opinion would most assuredly have forced it, especially coming on the heels of Pearl Harbor.  As naval historian James Schlesinger explained, “[W]ithout Midway, there would have been no D-Day on 6 June 1944, with all that implies about the condition of postwar Europe.”[xvi]

The Combat Intelligence role in Midway would remain mostly a mystery following the victory.  The division between Hypo and Washington offices hardened. Washington’s initial hesitancy and differing conclusions proved foolish and inaccurate.  The personal bitterness of the Redman brothers and their team in Washington against Rochefort in particular erupted shortly after Midway.  The man most responsible for intelligence success, and perhaps the Pacific theatre’s most experienced analyst, Joe Rochefort, was abruptly transferred out of Pearl.  Subsequent attempts by Layton and Nimitz to have the Navy Department officially recognize Rochefort’s contributions was met with criticism and disdain, mostly at the hands of the Redmans.[xvii]

It would be up to historians and some of Rochefort’s most ardent supporters to rectify the lack of recognition.  As information slowly became declassified, Rochefort’s role in the Midway victory became clearer.  “For World War II buffs, Rochefort [had] long been a cult hero – the maverick who wouldn’t play the Navy’s bureaucratic games, who wore a smoking jacket and slippers on duty for weeks on end in a cellar, who insisted on the integrity of the intelligence process, [and] who was ultimately proved right.”  A 1942 Top Secret evaluation of radio intelligence in Pearl had determined that due to the successes of Rochefort and Hypo, Midway was “all over but the shooting” once Rochefort had broken the Japanese intercepts.[xviii]

While no single person, or event, can ever really be responsible for victory in such a large and complex military engagement as Midway, the success of Rochefort, Station Hypo, and the intelligence analysis played a vital role in the stinging defeat of the Japanese fleet.  As Layton observed, “For a week in May, the entire course of the war turned on the outcome of a struggle fought by cablegram and intercepts within the Navy Department”.  In 1986 Rochefort would receive a posthumous Distinguished Service Medal “for playing the key role in winning the decisive battle of the Pacific war.”[xix]

            The war in the Pacific would rage another three years.  Nimitz would become one of the great naval heroes of World War II.  His rival Yamamoto would meet his death in April of 1943 at Solomon Islands when yet another JN-25 decrypted message would prove his undoing.  Some of the code-breaking success of the Navy would leak out, notably in the Chicago Tribune, and the Japanese eventually realized their codes were no longer secure.  Yet cryptanalysis and code breaking found further successes in the Pacific and European theatres.  The legacy of Midway was as much a human triumph as it was technological.  Without Rochefort’s knowledge and skill in cryptopgraphy and Layton’s determination to force the issue of the analysis, “AF” might have remained a mystery incorrectly identified by the Washington department as another location, and Midway might have fallen.  Without the trust in Hypo and the personal connection between Nimitz and his Pearl Harbor intelligence office, Redman and his office might have steered the Pacific fleet into another disaster such as the one on December 7, 1941.  Certainly it took the men flying the bombers, and manning the carriers and ships that made up the Midway task force to win the day, but in the heat of the many firefights those men owed a great debt to a man in a basement at Station Hypo without whom they may not have survived.