API limitation

[Qyu]

I have a script which return some informations about players and games

But since some website update, i discover there is now an API limitation :

You have made too many API Requests recently. Please leave at least 10 seconds between API requests.

What is this limitation ? how many API request can we do without having it ?

[dgz345]

5 every 10 second.

[Qyu]

Thanks

does the limit for all 'mode', even for the "mode=gamelist" or just for some ?

And any chance that the limit be deleted in the future ?
I explain : for example, when we want to have some players information, we could use the API, but with a such limitation, the API is no more useful, and we should use the website html for get the information. And in the end, it's a lose-lose : more charge for the server for create heavy html page instead of light xml, more work for us for parse the information.

[dgz345]

yea i know.
going to say this. as a tech team the only thing is that you moderates the bug and tools forum and being part of the tech discussions. so i dont know what the admins are doing.
but i know bigWham is working on performance issues. he told me that some users sends 10000 of requests to the api. and around 30 per second.
if i was in charge i would remove the limit after the performance issues have been fixed

[Qyu]

Ok

Have you imagine a rolling cache, with a ttl of 24h, for the API ?
If a new API call is made (same GET value) is made, then a new cache is done (or if this one is older than 24h), ortherwise, just return the cache. Maybe add an hack, just for the API call of the Speed Turn extension (and/or other trusted extensions).
Then, add in the xml the timestamp of its generation, for information. I guess most of extensions users will be accommodate themself with that. If it's only for stat (for my case), having unfresh informations is not a real problem.
And with a such cache system, since the database use will be reduce, maybe add an option in the gamelist mode, for having all game, and not just by group of 200. Nevermind the cache is big, it's always better and easier for the server to send a single big datafile than a lot of smaller datafiles

[dgz345]

i dont have to much knowledge in cache and stuff. im a 19year student XD

http://www.conquerclub.com/api.php?mode=gamelist takes 7sec and returns every game.

[Qyu]

only the first 200
you have to add a GET variable : page=2 for the next 200, page=3 for the 401-600, etc....

[Qyu]

From scratch, here my 2 cents for a 24h rolling cache. it certainly can be improved
If a such improvement already exists, forget about it.

You add a boolean GET "cache", if cache=1, then you call the cache file, otherwise, you keep the current script (with or without the current limitation)

in the api.php

Code: Select all

$cache_name = md5($_SERVER['REQUEST_URI']); // so each type of api call has is own cache file, depending of the GET values (1)
if ($_GET('cache') && file_exists('cache_dir/'.$cache_name))  { // if the user accept the cache and if the cache file exist
 include('cache_dir/'.$cache_name); // return the cache file content
}else{
 /*** here the current api.php which return also the generated xml ***/
 $fp = fopen('cache_dir/'.$cache_name, 'w+'); // create/open the cache file
 fputs($fp,$xml); // write the cache file content
}


and you add a cron job, which delete all cache file older than 24h, via a script bash




(1)
It can be improved there, as the name will depend of the GET values order, and have two different cache file, even if the returned xml is the same (ex : api.php?cache=1&mode=gamelist and api.php?mode=gamelist&cache=1)
i also thought of a RegExp that create the file name by deleting all forbidden special character in file name from the REQUEST_URI, but it could be a problem for two players named ' foo?bar ' and ' foo/bar ' which give the same name. Anyway, a RegExp should be better