Focusing infinity with 12-50mm. Is it possible?

Hi all,

last night I tryed to take photos to milky way.

I was in a place very far from any light, so all was dark but starred sky.

I used my omd-5 with panasonic 20mm and I started to play.
I took several photos changing iso, shutter speed, aperture..

Due to bandings problem of omd with this lent I decided to change it and I mounted the 12-50..

...so,
camera is on the tripod
set aperture 3.5
set time 20 seconds
set iso 6400
set manual focus, and rotate ring to infinity..
click

nothing is focused

I was confused, I spent hours changing all shot settings, trying closing aperture, changin isos, disabling IS or whatever, but never was in focus.

With the 20mm I get focus in few seconds..

So I look on internet and I found some starred sky photo taken with the 12-50mm (so is it possible!)

The night later I spent 1 more hour without any result!

What's the problem?
Where am I wrong?
Lent seems perfect in any other "normal" shoot I took.

Maybe it was so dark that lent missed its focus? (It was very dark ? can't see my hand without a pocket torch)

Help!
G.