The 'h' (lower-case) stands for 'hybrid model'. Transistor amplifiers may be modeled as 2-port devices with 1 port being the 'input' and the other port the 'output'.
Any 'black box' 2-port device has sets of parameters that can be used to describe the electrical characteristics of what's inside the box. One type of these parameters is called the hybrid set, or h parameters.
h11 is the short circuit input impedance (Vin/Iin @Vout=0)
h12 is the open-circuit reverse voltage ratio (Vin/Vout @Iout=0)
h21 is the short-circuit forward current ratio (Iout/Iin @Vout=0)
h22 is the open-circuit output admittance (Iout/Vout @Iin=0)
For the common-emitter bi-polar junction transistor amplifier h21 is called hfe. 'h' is for hybrid, 'f' is forward, 'e' is for common Emitter.
The difference between hFE and hfe is: FE is for a fixed DC bias and a fixed DC current gain (and that must be specified for each value of hFE). hfe is the small signal AC current gain (and it is also specified for a given bias). hfe is frequency dependant. hFE is only good at DC.