Real Madrid’s club record 22-match winning run was ended, as Valencia came from behind to secure a famous victory.
Cristiano Ronaldo’s penalty put Real in front, his 33rd goal of the season.
But Valencia made it 1-1 seven minutes into the second half through Antonio Barragan’s deflected shot.
Otamendi then powered the ball in from a corner but Madrid stayed top of La Liga thanks to Barcelona’s 1-0 defeat to David Moyes’s Real Sociedad.
It was Real’s first defeat since they lost to city rivals and La Liga champions Atletico Madrid on 13 September.
The result lifted Valencia, beaten just once at home all season, into the Champions League places.
Carlo Ancelotti’s side would not have envisaged that scenario when they took an early lead through Ronaldo.
After Gareth Bale had been fouled, a free-kick from Tony Kroos was flicked on by Bale against the arm of on-loan Manchester City striker Alvaro Negredo.
Ronaldo converted from the spot but then missed an excellent chance to double Real’s lead when he rounded goalkeeper Diego Alves but, from a tight angle, shot into the side netting.
Valencia had five players booked in a niggly first half but struck a post through a deflected Andre Gomes shot in stoppage time.
And they were the better side in the second half, levelling when former Liverpool player Barragan scored his first league goal for the club with a shot which was diverted past Iker Casillas by defender Pepe.
Bale was then denied by an excellent Otamendi tackle at the other end before the Argentina international won it with a towering header with 25 minutes to go.
Negredo impressed for Valencia, who could have spared themselves some anxious late moments, notably when Isco’s close-range header was saved by Alves, had Barragan not wasted another excellent chance when he shot wide with just Casillas to beat.
Holders Real will now be looking to bounce back against their local rivals in the first leg of their Copa del Rey last-16 tie on Wednesday night, when Fernando Torres could make his first appearance since re-joining Atletico on loan.