Python is a high level scripting language that offers a simple syntax but has the versatility of a more complex language, such as C.  Recently it has also become a platform for scientific computing, offering an environment similar to Matlab but with much faster execution. 

Further information can be found at Python's main site, and the scientific computing information can be found at SciPy's main site.

Installing Python: (Windows Users)

• Download Python here.
  For scientific programming you also need Numpy and Scipy.  Matplotlib is useful for creating graphs.
• Install the binaries and you're set!
• The base installation comes with an Integrated Development Environment (IDE), however I like PyScripter.  There is only one problem, Scipy's Weave module (which allows the use of inline C to speed up execution) does not compile properly.  To fix it include this code in your modules:

• import sys
• sys.execution = 'C:\Python25\python.exe' (or wherever you put the python exe file)

More advanced Python (Windows):

In order to compile libraries, modules, or use Scipy's inline C-code you need to install the gcc compiler and a Unix emulator.  MingW and Cygwin are both good choices.  MingW is the minimum you need whereas Cygwin offers a "full" Unix environment on your PC. 

The easy installation for MingW is here.

Cygwin (and MingW) installation instructions are here.

Python on Mac OS X: 

Python comes preinstalled on 10.5 (Leopard) but is slightly out of date (2.5.1 vs 2.5.2).  The new version is available from the downloads page.  

The main difficulty in installing SciPy and NumPy is building the algorithm libraries (LAPACK).  If you have 10.5 and are not sophisticated, a Universal Binary for all the necessary scientific computing modules is the SciPy Superpack.

After both of these are installed, you are good to go :)

Getting Started:

Python has a decent tutorial available on their website. 

The Scipy documentation includes tutorials, tutorials for Matlab users, and tips on optimization.

Examples comparing Python and other implementations (here and here).

Additional Resources:

StatPy: Collection of resources for statistical computing in Python, including R/S-Plus interfaces and modules/extensions to optimal libraries.