覆盖magento搜索系统以允许随机顺序

I need to find a way to either add another sort order so that I can sort by random or I need to override the core search system to allow me to achieve the same. (or some combination of the two?)

I know that the magento search files are in app/code/core/Mage/CatalogSearch/ but I am unsure as to which files I need to look at changing.

The file you are looking for is app/code/core/Mage/CatalogSearch/Model/Layer.php. The name of the class is Mage_Catalog_Model_Layer. Inside it you see the function prepareProductCollection($collection).

I suggest you rewrite this model (see this or this) and override the aforementioned function and add/implement you sorting logic. An example of overriding this function would be

class XXX_XXX_Model_CatalogSearch_Layer extends Mage_CatalogSearch_Model_Layer
{
    public function prepareProductCollection($collection)
    {
        parent::prepareProductCollection($collection);
        $collection->addAttributeToSelect('some_attribute');
        $collection->setOrder('some_attribute', 'asc');
        return $this;
    }
}

go to file location community\app\code\core\Mage\CatalogSearch\Block\Result.php and find function public function setListOrders() and here you can change to your custom order