terminal git push slow

I'm not sure what's happening. I've always done my git commit and push from terminal. Sometimes git commit from PHPStorm (OSX) and push from terminal.

My SSH key has a passphrase.

Lately, I decided to get fancy and do my git push from PHP Storm, and it works without any problems. The first time, it prompted for my pass phrase. I entered it. Saved it (by the way, I couldn't find where to delete saved pass phrases). 

Now my issue is when I go back to terminal to do a git push. It is extremely slow. It takes around 3-5 min. to prompt for the passphrase. 

To test. I restarted my system. Did everything in terminal. Not a glitch. When I start PHPStorm again. The git push from terminal goes back to 3-5 min. 

I have absolutely no clue what is going on. I don't even want to fix it at this point. I just want to unhook PHPStorm from my SSH key or whatever it's doing and go back to terminal. 

Thanks,

JG