Pointing to http://www.octave.org is good, because that gives people a direct way to find out more. If citation of a URL is not allowed by a publisher, or if you also want to point to a traditional reference, then you can cite the Octave manual:
@BOOK{eaton:2008, author = "John W. Eaton and David Bateman and Søren Hauberg", title = "GNU Octave Manual Version 3", publisher = "Network Theory Limited", year = "2008", isbn = "0-9546120-6-X" }