Well, you'll get better at that with time. You should be building those skills anyway if you want to be a serious musician, at all.
And who knows, maybe it'll end up different from what you originally had in mind, and it will be even BETTER.
I don't usually jot my ideas down in perfect pitch, either. (Because I don't have perfect pitch. I'm CLOSE, in that I kind of always know what C sounds like, but that's is; not perfect. And usually a little flat.)
So, I either ASSUME I'm in the key of C in my head, and then just write down note names like C D E F# E D G, or I do it with scale degrees, like 1 2 3 #4 3 2 5.
And there's shorthand you can learn for notating rhythm. I usually just write the stems and beams, and leave out note heads.
So my notes look like this:
C D E F# G F# E D G C
| |-| | = | = | = | |-| |
Or at least that's the best I can do typing it in.