old-school shooters relied more on enemy placement and level layout in order to make up for their simplistic AI behavior, it's why I believe Doom getting criticized for 'simple' AI is a moot one, too many people use AI as an argument without considering the ramifications different AI could have on the gameplay itself
a shotgun zombie on his own isn't much of a threat, but when he's placed at X interspersed with Y enemies in a Z kind of room you have your challenge cut out for you
Serious Sam does this by massively increasing the enemy numbers and enemy composition in groups (which enemies are used) to the point where obstacles and level layout play a lesser role because it's not as possible to handle such a large amount of enemies in smaller spaces
the increasing complexity of AIs over time also led to fights against enemies being a challenge of their own rather than the challenge resulting from a multitude of variables such as enemy count and level layout (although they can still play a large role to some degree), like in Unreal Gold where the enemies you usually fought almost functioned like multiplayer bots, who possess degrees of skill the player can have too, like dodging attacks and leading shots, however level layout played a lesser role for combat in comparison to other contemporary shooters
alternatively, 'good' AI can result from a need for better pathfinding, such as in Descent or F.E.A.R, where enemies had to be able to move in full 3D space much like the player could in Descent on top of giving each enemy type unique behavior, and enemies in F.E.A.R had to be able to use the environment and the ability to flank much like the player can do the same against the enemies
by now you should have noticed that one element which many games acclaimed for their enemy AIs have is the ability for enemies to be on an equal ground with the player in terms of skills, they should be able to play the game much like how you can, same shit goes for real-time strategy games, fightans and multiplayer bots
older shooters would rely on numbers and placements for challenges, but the advent of better AI makes fighting one enemy with good AI already challenging, whereas the effects of AI on individual units become less noticeable the higher the enemy count gets (how would people even propose better AI for Serious Sam? what is a headless kamikaze or Kleer supposed to do but run at you and kill you up close?), unless we're talking herd AI which in turn makes enemy types less relevant due to them no longer be as predictable and constantly breaking established rules as opposed to F.E.A.Rs unpredictable clone soldiers
one example of outright inept AI would be DOOM 2016, where the player can use vertical geometry to his advantage and outrun enemies, whereas the enemies are wholly uncapable of using vertical platforms to the same extent the player can, resulting in vertical platforms being more of a safe zone for the player as the enemies can only play catch-up (another bad way of going about vertical space is what Shadow Warrior 2 did, where EVERY enemy can jump up any ledge and vertical space in order to catch up with you, turning vertical space in combat to an irrelevance rather than an asymmetrical advantage)
on top of that, the enemies are not particularly challenging to fight on their own as they cannot dodge or utilize geometry to the same extent you can, and with no level layout and enemy placement to make up for it, resulting in boring samey encounters
defense in nuDOOM also largely comes down to circlestrafing, if you're circlestrafing and jumping about you have practically reached the skill ceiling when it comes to avoiding damage in nuDoom whereas enemies in games like Descent 2 would actually lead their shots on higher difficulties, forcing you to move erratically rather than methodically (something not always easy to do given the tight amount of space in Descent levels!)
in short, I believe levels were as complex as they were in older shooters in order to make up for simple AI, as otherwise fighting a bunch of simple enemies on flat maps like in Wolfenstein 3D wasn't all that fun
newer games tend to take the Serious Sam route by compensating for level complexity with enemy amounts (rather badly when it comes to enemy composition, if I might add), as games with outstanding AI and levels to take advantage of that are a rarity due to the time and effort it costs