In the early 1990s, Microsoft Corp.’s Windows operating system was dominant for office applications but a loser in the world of computer games.

Windows inserted itself between game programs and the computer hardware in a way that slowed down graphics and animation. Game developers vastly preferred the DOS operating system, which didn’t gum up their special effects.

That created an opportunity for three Microsoft misfits— Eric Engstrom, Alex St. John and Craig Eisler.

Mr. Engstrom, who died Dec. 1 at the age of 55, and his pals formed one of several factions within Microsoft trying to solve the game problem. Openly contemptuous of colleagues who didn’t share their ideas, they were so obnoxious that Brad Silverberg, who ran the Windows business, dubbed them the Beastie Boys. He had to fend off frequent demands for their dismissal.

Yet the solution they developed, DirectX, beat anything else on offer inside Microsoft. DirectX software recognized games and allowed them direct access to the computer’s graphical capabilities, allowing a richer game experience than DOS could.

WSJ Obituaries

“It was brilliant,” Mr. Silverberg said. Launched in 1995, DirectX wowed game developers and led to a flood of new games for computers loaded with Windows. That success emboldened Microsoft to plunge deeper into the lucrative gaming market by developing the Xbox console. Microsoft’s game business produced $11.6 billion of revenue in the year ended June 30.

Mr. Engstrom, a self-described bumpkin, celebrated his success by buying four Porsches, a custom-chromed Harley motorcycle and a waterfront home in Kirkland, Wash. He took his children to Disneyland or Disney World annually.

He died at a hospital in Seattle. His wife, Cindy Engstrom, said he had injured one of his feet in October, accidentally took too much Tylenol for pain relief and suffered liver damage.

In recent years, Mr. Engstrom worked as a consultant and pursued various business ideas. He also was working on ways to make better drones and had detailed ideas about how to generate power from cold fusion. Confronted with any technical challenge, he typically asked, “How hard can it be?”

George Eric Engstrom was born Jan. 25, 1965, and grew up in Oroville, Wash., a small town near the Canadian border. His father had a store selling boots, saddles and other Western gear.

As a child, he wore an astronaut costume and gazed at the family TV in 1969 when Neil Armstrong walked on the moon. Before long, he proclaimed his plan to be the first man on Mars—an ambition he never renounced.

Helping out in his father’s store was too down-to-earth. He loathed chitchat and the idea that customers were always right. “I have spent a lot of time in life dealing with the public, and I have frankly liked very little of it,” Mr. Engstrom said in a 2003 video interview at the University of Washington. “People are unpredictable to me, so it is always very hit and miss whether I can find a way to communicate with them.”

During his teens, he earned money by programming computers for local customers. At Washington State University in Pullman, he completed only a few courses. He worked as a freelance programmer, then moved back to his parents’ home in his early 20s after losing an important client. For a time, he ran a carwash.

Then a friend urged him to apply to Microsoft. Mr. Engstrom showed up for the interview in a new plaid shirt and pressed jeans. “I dressed the way any country bumpkin would dress,” he said later. Even so, he got a short-term contract for customer-service work, requiring him to teach himself the Fortran computer language.

After his temporary job ended, Mr. Engstrom had offers for longer-term work at both Microsoft and Data I/O Corp. Data I/O offered a higher salary, but Microsoft dangled stock options. He chose Data I/O because he thought options were something offered “to people in plaid shirts to confuse them about cash,” he said later. “That was the first time I lost $10 million—and unfortunately not the last.”

He returned to Microsoft in 1991 and worked in a unit providing technical tools to software developers. Messrs. Eisler and St. John invited him to join their weightlifting sessions and persuaded him to work on game technology.

“I don’t like games, so I had no idea games were huge,” he said later. The trio dubbed their work on DirectX “the Manhattan Project,” a nod to what they saw as its importance and top-secret nature.

After DirectX, Mr. Engstrom worked on an ill-starred web-browser project and was a witness for the defense in a 1999 hearing during the Microsoft antitrust trial. His brash style served him well during testimony.

A government lawyer asked Mr. Engstrom if he had ever said Microsoft wanted to control certain software tools “down to the iron,” according to “Renegades of the Empire,” a 1999 book by Michael Drummond. “That would not surprise me at all that I said that,” Mr. Engstrom replied. “I was quite full of myself at that point in my career.”

He temporarily left Microsoft in 2000 and founded Wildseed Ltd., which provided ways to customize cellphones with different looks and graphical capabilities. That concept fizzled out, and America Online Inc. bought Wildseed in 2005. Mr. Engstrom worked for AOL before returning to Microsoft.

With a lifelong friend, Swain Porter, he also set up Catalytic Software to provide software-development services at a planned 500-acre company town called New Oroville (a homage to their hometown) near Hyderabad, India. They built white domed houses, resembling giant space helmets, for employees. At one point, a water buffalo toppled into a trench dug for fiber-optic lines and had to be extracted from the mud. Catalytic ended up with a smaller, 50-acre campus and was rocked by the 2008-09 financial crisis. The business wound down in 2010, and the assets were sold.

Mr. Engstrom met Cindy Smith at Wildseed, where she headed marketing. They chose Groundhog Day in 2005 for their wedding, partly because they thought it would help Mr. Engstrom remember the date. Still, he forgot their first anniversary. He is survived by his wife and four children.

He wasn’t deterred by business setbacks, friends said. “He thought things were possible that nobody else on the planet thought would be possible,” said Ben G. Wolff, a friend who runs a robotics company, “and sometimes he’d be right.”

Write to James R. Hagerty at bob.hagerty@wsj.com