Need a guide to math software.

(what passes for credentials: D.Phil in theoretical physics, particularly numerical simulation, and undergraduate physics teacher for a few years)

MatLab and Octave are very similar (Octave is an open-source version of MatLab); they're simple programming language that specialize in numerics, particularly matrix manipulation. Neither, last time I used them (I taught a course in computational physics using matlab and tried to run student code in octave) is particularly strong with symbolics. In other words, if you want a number for an answer, they're okay...

...but honestly if you want a number for an answer, I think as a high-schooler you'd be better off learning Python and using the Numpy/Scipy libraries. That way you have a real general-purpose language to use for lots of other things. Matlab and Octave are okay, but they're terrible programming languages (I did my thesis research in Python/numpy; it's very capable).

For symbolic math, Mathematica is the commercial standard and a lot of people swear by it. For high school math where you're typically doing fairly straightforward symbolic work ("Integrate x^2 + 2x+ 1"), something like Mathematica is useful...

...but for whatever reason, I personally can't stand it and wound up using Maxima (http://maxima.sourceforge.net), which is an OSS symbolic math tool. It's not the easiest thing to learn, but if it can handle multipole expansion of tensors for me, it would do just fine for a high-schooler, plus would cost them nothing so they could use it for life.

Python has a symbolic library too (SymPy) but for just playing with concepts, I still recommend Maxima, particularly with one of the great front-ends (I personally use Emacs with iMaxima (https://sites.google.com/site/imaximaimath/), but I'm not sure I'd saddle a high-schooler with emacs unless they were really into computers).

So my recommendations: Python/Numpy for numerical work, Maxima for symbolic work. Good solid tools, multiplatform so good on any computer and free to boot. Just what high school needs.

A lot of math teachers in the twitterverse like brilliant.org
From their website: Brilliant.org provides a challenging intellectual environment to anyone with internet access and a passion for mathematics and physics. By connecting motivated thinkers worldwide, Brilliant.org aims to be a hub where the worlds most promising minds recieve the opportunities they deserve, no matter where they are from.

An open source competitor to MATLAB / Octave that has gained a lot of traction this past year is Julia. Julia performs at speeds of C/C++ with a syntax that is similar to MATLAB. Julia also has a package management system similar to R but, for now at least, with far fewer packages than R.

Learn more about Julia here: http://julialang.org.
An IDE for Julia is available here: http://forio.com/julia
Discussion the language here: https://groups.google.com/forum/#!forum/julia-users

I would take a look at Sage (http://sagemath.org). It's a Python-based system that combines a lot of free and open-source math software (including SymPy and NumPy). The graphical interface runs in a web browser. You can try it without installing anything.

Beware anything commercial: if your highschooler has access to such software, can you guarantee he or she will continue to do so? Imagine going to college and no longer being able to use the stuff you've gotten used to. Free software can't be taken away, so if you learn how to use such a tool, you can take it with you wherever you go.

I use Mathematica exclusively. That said, I have seen a free package called Geogebra (http://www.geogebra.org) which looks extremely impressive; I just haven't taken the time to learn it. Were I in high school and had no access to Mathematica, I would give this a try.

Perhaps I misunderstood the question, but it seems to me that the asker wants to boost her highschooler's MATH skills not teach him a math PROGRAMMING language. If that is the case (I am a Learning Skills Specialist in math and chemistry at our local Community College), I suggest trying some free websites like the DESMOS graphing calculator and Wolfram-Alpha which will answer almost any math problem (though not always in the format I'd like when you get to Calc II and parametrics/polar coordinates). If I misinterpreted the question, please forgive my ignorance....

I previously reviewed an algebra app; have you tried it? http://kk.org/cooltools/archives/6504

To the MATHEMATICA users here: is there an online course, book, or website that would help a teacher or student use Mathematic to help learn mathematics? Let's imagine it is a self-guided course. Let's say I want to brush up on my mathematic skills in the evening. Is there interactive software course for such a thing?