Map Cache

[xevic]

Hi developers, we notify in our ct wild server some problems with maps downloads, special in high pinged users as it's logic, and this make me think that everytime a map has to been played, it needs to be dowloaded again.

Can a caché system be done in a next version? looking 4 a map on local resource directory, i know they are downloaded there, then using them could help server to save a lot of bandwidth and help to avoid lag or retards on start levels with maps.

It can be done with just a name check, but a signatura check will be better, special to avoid bad map makers that doesn't change version of the file on a edit, sending a couple of strings in place of all the xml when it is in the hdd will be cool

It will be cool it will be added in 2.8.3 for example, not just in 0.3, cause it isn't enough stable to a permanent server.

If a map is just downloaded 4 who hasn't it it will give a lot of performace on servers with maps.

Thx 4 ur atention.

ps: excuse my spaninglish

[Tank Program]

I thought it already did download to the local repository?

[Z-Man]

We already have a cache, it's in resource/automatic. It's name based, that's enough because the book of rules says the version and hence the name of a map needs to be changed on every content change. Of course, downoloading a new map still lags you in the sense that ping measurements shoot up; the client is completely unresponsive during the download. That can mess up the next round. We now have limited multithread stuff going so waiting for your video card to swap doesn't block the network, I'll see whether that can also be used during resource downloads. Of course, that won't help if the map download fails or takes too long.

[xevic]

ok, thx bad luck to high pings players, jjjj.

[philippeqc]

Hi developers, we notify in our ct wild server some problems with maps downloads, special in high pinged users as it's logic, and this make me think that everytime a map has to been played, it needs to be dowloaded again.

I'm not sure about this, but MAYBE it could help if the map are served by a different computer or from a different netword (ie: not another computer sharing the same line to Internet). Do you have your maps on the resource repository? Have you tried to put them up on a geocity (or what ever hips kids use today)?

Can a caché system be done in a next version? looking 4 a map on local resource directory, i know they are downloaded there, then using them could help server to save a lot of bandwidth and help to avoid lag or retards on start levels with maps.

As explained by z-man, the map are already cached. Sadly, new clients still need their copies. And will need to attempt to download it for every single map. One capacity of the resource manager that is wildly under used is the ability of user to manually download and install maps in their local repository. If you where to provide an archive containing ALL the maps of your server, someone could easily download it, and install it locally and avoid the downloads between rounds.

So what happen when you release new maps, well simply, the same thing as before. But I'd guess that maps might have a tendencies to come in clump. Its not one new map, its 3 new maps, with modification on 2 existing ones. That might be wortky of a common archive. ;)

It can be done with just a name check, but a signatura check will be better, special to avoid bad map makers that doesn't change version of the file on a edit, sending a couple of strings in place of all the xml when it is in the hdd will be cool :D

If the map designer updates his creation and forget to modify the version tag AND the server admin manually load the file on her server AND didnt think it special that his configuration files /rotation files dont need to be updated, I'd call for criminal negligence. And seeing how we could all use the entertainment, I'm quite sure I wont get much opposition if I propose public flogging as an adequate punishment for such a hienous crime.

So in short, look into
a) Hosting on a different server / faster server
b) making archive of all the maps of your server (*)
c) Before adding a map to the server, have a quick look at it and make sure the version is update. Or ELSE!!!!!!

-ph
(*) Or all the maps on the resource repository. Hint hint!

[Luke-Jr]

As a CTWF player, I noticed the complaints of the problem starting approximately at the same time as the server began using a special CT repository instead of the official one.

[ed]

As a CTWF player, I noticed the complaints of the problem starting approximately at the same time as the server began using a special CT repository instead of the official one.

We were getting the problems while using the official repository. Which is why we started using a CT one.
The switch happened to coincide we the introdution of 12 new maps to the server.
There's got to be around 50 maps in use there now.
It seems ok again now, as most regulars have the new maps locally.

Personally I've never had a problem there, and I always empty my local map resource when I upload new maps to make sure it's working correctly. But I get a lot of reports of players being kicked off the server at the start of a new map round. They then reconnect, the map downloads and they can watch the map play and all is ok again. It happens more on high ping players.

It seems their PC can't download the map in time to start the round, so it boots them out. This is a guess on my part.

Couple of possible solutions.

Don't start the round till all players have downloaded the map.

When a player first connects to the server, if he doesn't already have them locally, have him download all the maps in current use before he can join play.

[Lucifer]

It's the addition of the maps that's the problem. I noticed it myself a little while ago.

The problem goes like this:

I connect to the server, and I don't have the map, so my client downloads it as expected.

For some reason, it's going slow. Maybe my own connection is being used a bit, maybe there are several people downloading it at once and your server's being used a bit.

No problem, the download is finished before the round ends.

Next round: new map. Only this time my client doesn't even know to download it until the match is about to start.

There are a couple of solutions to this problem, some that we can do, at least one that you can do.

What we can do:
1. Make the client go into spectator mode to download a map. This requires the server to keep track of which team he's on so he can be returned to that team after the map download is finished.

2. Make it possible to provide a list of resources for the server that the client can download and cache at will, hopefully it'll get enough of them cached to avoid problems.

What you can do:
Change your map rotation script so that it sets MAP_FILE right after the round starts. That way, the client receives the change right away and can start downloading it. Actually, I don't think this will work without changes in the code, but it's a pleasant thought.

Provide all the maps in a zip file for someone to download, instruct them to unzip it into their ~/.armagetronad/resource/automatic directory, and make the zip file start there and contain the correct directory structure. That way a person can manually update their cache.

[Luke-Jr]

We were getting the problems while using the official repository. Which is why we started using a CT one.

Well, then don't work around the problem (by starting a new one)-- report it =p
I've never had any problems with either-- and I don't expect I'd ever have problems w/ the official repository since it's on my LAN, so these things need to be reported or they'll go unfixed.

Don't start the round till all players have downloaded the map.

We already can pause the game for someone chatting, so I imagine this should be possible... if the map downloading affects ping response time, then perhaps send the map, send a ping, then wait for all players to respond to the ping before starting the round...

When a player first connects to the server, if he doesn't already have them locally, have him download all the maps in current use before he can join play.

You need maps to watch, too.

[ed]

Well, then don't work around the problem (by starting a new one)-- report it =p)

I first wanted to see if the problem was with your repository, or with map repositorys in general. Seems to be the latter.
Consider it reported.

[Z-Man]

We're already waiting for clients to finish downloading and doing other stuff; try increasing GAME_TIMEOUT. The fundamental problem, that AA's own networking system is totally blocked during resource download and that the server has no way to tell the difference between a client that's downloading and a client that just crashed, won't go away.

Why don't we push all resource fetching into a background thread? If the interface section between the resource thread and the main thread is lean, and it is if the background thread just fetches files and tells the main thread when they're ready, multithreading issues are very well controllable. We could then introduce RESOURCE_PREFETCH, a server controlled variable that tells the client which resources it should prefetch if it has nothing else to do. This could hold all the maps the server uses, or smarter, the upcoming maps of the next few rounds, because otherwise the resource server will be under heavy strain from players who enter the server for the first time and get told to fetch everything.

[Luke-Jr]

Why don't we push all resource fetching into a background thread? If the interface section between the resource thread and the main thread is lean, and it is if the background thread just fetches files and tells the main thread when they're ready, multithreading issues are very well controllable.

However, what does the main thread have to do while the fetching is in progress? All it can do is tell the server "hold on, I'm downloading", which adds to the bandwidth consumption if done too often...

We could then introduce RESOURCE_PREFETCH, a server controlled variable that tells the client which resources it should prefetch if it has nothing else to do.

Instead of trying to put all resource fetching in a separate thread (and thus require resource-using code to be non-blocking), why not have RESOURCE_PREFETCH alone be responsible for a new thread? A simple flag (mutex? not familiar with threading terms, sorry) can be used to force only a single download at a time, with some additional code to pause prefetching when something else is more important.

This could hold all the maps the server uses, or smarter, the upcoming maps of the next few rounds, because otherwise the resource server will be under heavy strain from players who enter the server for the first time and get told to fetch everything.

We can also work on ensuring mass resource downloads aren't a problem. I'd really like to get some kind of XMPP pubsub setup so we can have live mirrors of the repository and thus round-robin capabilities.

[Z-Man]

However, what does the main thread have to do while the fetching is in progress? All it can do is tell the server "hold on, I'm downloading", which adds to the bandwidth consumption if done too often...

While some important resource is getting fetched, yes, that's all it would do and that's more than what it does now, play dead. Actually, it will not send needless pings, but it will respond to server messages (console output, setting changes) with acks that tell the server the client is still alive. No additional bandwidth used. The real benefit would be the background prefetching of resources, the game can run while that's happening. Provided we enable the resource system to throttle the download.

The resource system basically is something right now that you pass a file path and get a full filename or FILE back, right? No threading trouble then that the caller/resource user has to worry about, it's the task of the resource system to only let that function return when the resource is completely fetched.

[Luke-Jr]

The resource system basically is something right now that you pass a file path and get a full filename or FILE back, right? No threading trouble then that the caller/resource user has to worry about, it's the task of the resource system to only let that function return when the resource is completely fetched.

That's blocking I/O, yes. The resource fetching can't just go start a new thread and return, the caller would need to be aware of the wait.

[Z-Man]

Right, and what's the problem with that if the resource is really needed right now, like the maps are? The caller can choose whether to use this blocking path and a to-be-implemented non-blocking "Get me this resource if available, if not, start to fetch it, possibly tell me when the fetching is done" path and get along without the resource while it is unavailable, like replace the custom bike texture with the default texture. In the blocking case, it has to give a hint whether game network activity is to be allowed while the download is going on.