By default any robots.txt file present on the server is ignored - this is because it's expected that robots.txt would be relevant only to external 'bots'. To have robots.txt not be ignored, set Configuration.RespectsRobotsTXT to true (see section on configuration for more).
The robots.txt must be in the root of the web-site (eg. domain.com/robots.txt), per the 'robots standard'. The engine looks first for instructions specific to User-agent: Keyoti_Search, and then to any wildcard User-agent instructions.