VAS usage issue ...

Any issues, problems or troubleshooting topics related to the Prepar3D client application.
User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Sat Jan 04, 2014 7:57 pm

Kakusso,



FYI - I use Process Explorer (what you see in Fruggle's video) and FSUIPC memory monitoring. Process Explorer will only poll memory usage of a process every 500ms (1/2 sec) this is too slow for the rapid OOM that occurs in P3DV2. In the rapid VAS usage (OOM) problem, process explorer may only show 3.1GB VAS at the time P3D OOMs ... this tells me that P3D was attempting massive (1GB+) memory allocations which triggered the OOM.



On a side not, Fruggle doesn't explain where the chimes come from ... this is NOT a built in feature of P3D, it's a feature of FSUIPC (which you can turn on/off). Also, FSUIPC is an InProc DLL so if P3D OOMs then FSUIPC will not be working either (dependent DLL) and hence no chimes. I believe FSUIPC will start to chime when there is only 300MB VAS left ... so if P3DV2 is currently at 3.1GB VAS and then attempts to allocate another 1GB VAS, there will be no warning from FSUIPC.



Fruggle and Kosta are correct, moving from Airport to Airport consumes more and more VAS and if you happen to fly over a major airport or city (even at FL330), it will consume more VAS and stay resident (in memory).



In C/C++ there is no "thread safe" way to do garbage collection and/or force release of allocations (just tested this recently), in fact, the more process intensive the task, the less likely a release is going to happen. Now, as I understand it, there is a non-thread safe way to force the release of memory allocations ... but it's NOT THREAD SAFE ... which means it could introduce all kinds of other errors/problems.



Rob
Rob Ainscough

Kosta
Posts: 1173
Joined: Sun Sep 18, 2011 10:20 pm

Postby Kosta » Sun Jan 05, 2014 2:45 am

Rob,

This is the same behavior we have in FSX, and P3D 1.4 and v2. Nothing changed here. What you describe is the problem of the app not freeing up the memory usage (VAS in this case) but keeps it occupied. This is why I described on my blog a very simple problem, when you fly from A to B, and you have C airport that you overfly on your route, the C airport will load but when you pass it, it won't unload. It will only partially, leaving VAS higher than it would be if C were deactivated. And this is exactly a problem, when you have more than one airport you fly over, eventually they will get the VAS so high up, you won't be able to land at your destination airport.



Additionally, P3D v1.4 and v2 in a clean state, both start with higher VAS than FSX ever did (based on most comparative settings possible). And it will also up the VAS on each setting change, and not by a small margin.

simmerhead
Posts: 108
Joined: Sun Sep 25, 2011 12:25 pm

Postby simmerhead » Sun Jan 05, 2014 6:27 am

I can confirm the same problem. VAS is not freeing up as one would logically expect it to during flight.



I have also done extensive testing while changing display settings. If you increase settings to a high state, then go and change them back down again, VAS usages stays high as if you still run the sim with high settings.



With only 4GB of VAS available this problem needs some attention or else it will be very hard to use P3D v2 for complex scenery and aircraft addons, even though some work has been shifted to the GPU.

minime
Posts: 1198
Joined: Mon Jun 10, 2013 4:33 pm

Postby minime » Sun Jan 05, 2014 9:17 am

One thing to keep in mind... while this obviously could be a bug in Prepar3d, the memory is still managed by Windows. Which means the final control over what kind of memory is released and which is not lies in the hands of your operating system and not Prepar3d. So there is a chance that Prepar3d DID in fact give back the memory but that the OS did not claim it.



This is mostly true if a program is using managed code, for example in C#, which I know Prepar3d is using now. I do not know if any of _that_ code is managed now, but just keep in mind that it is impossible to make any assumption about memory allocation and bugs unless you have the source code of the program.



We know that Prepar3d has memory leaks, the question is just how many and how big these leaks are. Only Lockheed Martin can actually find that out, remote diagnosis is very hard.

User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Tue Jan 07, 2014 1:54 pm



Quote:

Quote from minime on January 5, 2014, 09:17

One thing to keep in mind... while this obviously could be a bug in Prepar3d, the memory is still managed by Windows. Which means the final control over what kind of memory is released and which is not lies in the hands of your operating system and not Prepar3d. So there is a chance that Prepar3d DID in fact give back the memory but that the OS did not claim it.



This is mostly true if a program is using managed code, for example in C#, which I know Prepar3d is using now. I do not know if any of _that_ code is managed now, but just keep in mind that it is impossible to make any assumption about memory allocation and bugs unless you have the source code of the program.




DX11 is not "managed code" - and as I understand it they are still using C++ for the core not C# (hopefully LM can confirm). I know they have SlimDX (you can see it in the deployment root) which is a wrapper around DX11 and is needed when using .NET framework managed code - However, that doesn't mean that unmanaged code can't be used and/or isn't being used in P3DV2. I'd be VERY surprised if entire P3DV2 is working under fully managed code.



There are many ways to remove memory allocation and one hope's LM isn't relying exclusively on GC in .NET 4. Note that .NET GC is not a feature of the OS, it's an implementation within .NET framework. The OS manages VAS ... you ask for memory the OS allocates it, you tell the OS your done with memory, it de-allocates it ... the GC (garbage collection) within .NET framework is simply a way to help speedup the .NET framework (a common complaint with developers that .NET framework is slow) by marking memory for de-allocation which can happen at a later time.



The .NET framework was never aimed at task intensive processing (such as a flight simulator) -- it's focus has always been business applications (not 3D simulations hence why DX11 is not considered "managed" and is NOT part of the .NET framework). It's rare that a business application to operate at such a high utilization (as we see in P3DV2) give plenty of time for GC to happen -- I doubt LM are relying on .NET GC for their core flight processing.



But as far as DX11, there are very specific calls to release resources.



Rob

Rob Ainscough

User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Tue Jan 07, 2014 2:42 pm



Quote:

Quote from Kosta on January 5, 2014, 02:45

Rob,

This is the same behavior we have in FSX, and P3D 1.4 and v2. Nothing changed here. What you describe is the problem of the app not freeing up the memory usage (VAS in this case) but keeps it occupied. This is why I described on my blog a very simple problem, when you fly from A to B, and you have C airport that you overfly on your route, the C airport will load but when you pass it, it won't unload. It will only partially, leaving VAS higher than it would be if C were deactivated. And this is exactly a problem, when you have more than one airport you fly over, eventually they will get the VAS so high up, you won't be able to land at your destination airport.



Additionally, P3D v1.4 and v2 in a clean state, both start with higher VAS than FSX ever did (based on most comparative settings possible). And it will also up the VAS on each setting change, and not by a small margin.




Hey Kosta,



I haven't noticed a higher VAS usage in P3DV2 as compared to FSX -- just the opposite, for example in FSX if I turn shadows on I take a significant VAS and fps hit and if I try to max out the water settings in FSX it will OOM almost instantly at any FSDT airport (even at a 6.5 LOD). If I dial P3DV2 down to close to an FSX equivalent (there really isn't an equivalent, more approximation) my VAS is lower. But I try to avoid comparing P3DV2 with FSX, many similarities but also many differences. I honestly think it is better to consider FSX a unique product and P3DV2 as a unique product.



What I have noticed is that 3rd party products VARY considerably in how they impact VAS usage in P3DV2.



But I agree that as one moves across the globe, one would expect objects that were loaded and are no longer visible (within whatever pixel threshold is set) that the object should be released and reduced VAS. My guess as to why they aren't doing this is "by design" ... I believe ESP uses a Quad Tree process of depicting the world in 3D space on a 2D monitor(s) ... navigating the tree and removing resource allocations no longer "in view" (pixel threshold) could be an expensive process in terms of CPU cycles and hence they just don't do it to keep performance high.



But, what I don't understand is that textures for compute shaders should live in VRAM (this is a big plus for DX11) along with the textures these compute shaders apply themselves to ... this doesn't seem to be happening in ALL cases ... it is happening in some cases because there is an obvious higher VRAM usage (I've hit 3.8GB VRAM usage, something that I'd never see in FSX) but NOT in all cases.



There most definitely is a VAS problem in P3DV2 that can be demonstrated on a clean install. 3rd party products are tending to exaggerate the memory management problems even more. But I think LM are aware of the issues and we can hope the next release/update solves many of them.



Rob
Rob Ainscough

User avatar
Beau Hollis
Lockheed Martin
Posts: 2127
Joined: Wed Oct 06, 2010 3:25 pm

Postby Beau Hollis » Tue Jan 07, 2014 5:03 pm

We did actually improve the releasing of resources in several places and fix some bugs that date back to ESP/FSX/v1 which improve VAS usage. Some of the new features use up more memory such as the tessellated terrain which doubles the texture usage because the night/emissive texture has to be stored separately to allow real-time lighting.



It's likely add-ons with many high res textures that are causing the biggest issues since we load autogen out at higher LODs much further away. We do still keep system copies of all textures around so that if a window is dragged onto another monitor hooked up to a second GPU, the textures don't have to be loaded from disk. We are looking into freeing the system-side texture memory after copying to the GPU which would likely help VAS usage a great deal at the cost of some extra texture loading in certain multi-gpu scenarios.



The reason for higher vRAM usage is render target resources for views, shadows, reflections, and post processes. Multi-sampling and HDR also increase the bit depth of render targets which takes more vRAM.



VAS usage for the terrain mesh and for shadow models would be significantly reduced in v2 though since we use the same model in the shadow view and the shadow views output to render targets on the GPU. Instanced trees should take up a bit less memory than batched trees did but then we also draw more of them.

Beau Hollis
Prepar3D Software Architect

minime
Posts: 1198
Joined: Mon Jun 10, 2013 4:33 pm

Postby minime » Tue Jan 07, 2014 6:47 pm



Quote:

Quote from robains on January 7, 2014, 13:54

There are many ways to remove memory allocation and one hope's LM isn't relying exclusively on GC in .NET 4. The OS manages VAS ... you ask for memory the OS allocates it, you tell the OS your done with memory, it de-allocates it ... But as far as DX11, there are very specific calls to release resources.

Rob




Yes, there are specific calls to release resources, but they do not work like you think they do.

Even if you free resources and use delete on memory that does not mean it gets freed by the operating system and reclaimed.



And that is on unmanaged code.



On managed code there is literally no guarantee whatsoever if and when memory gets reclaimed. But even in unmanaged pure C and C++ code, DirectX or whatever, there is no guarantee that memory is reclaimed by the OS and shown as free.



Here are a few things that can happen with memory:



http://stackoverflow.com/questions/17008180/c-delete-does-not-free-all-memory-windows



http://bytes.com/topic/c/answers/805894-delete-does-not-really-delete



http://stackoverflow.com/questions/1421491/does-calling-free-or-delete-ever-release-memory-back-to-the-system



So basicly, if you free memory, be it in C++ or C# managed or unmanaged, there is a certain chance that the memory is returned to the OS, but nothing more. On Solaris for example memory basicly never is returned to the OS, but even in Windows you can easily check for yourself that if you start fragmenting memory less and less will be returned to the OS.



So if you just let it run and do nothing, its perfectly normal that process memory gets larger over time.


User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Thu Jan 09, 2014 11:43 am

Very interesting read, but if memory isn't free'd up in any reliable manner, I honestly don't see much of a future for P3DV2 in a 32bit address space? I know you don't want to hear that, but I'm just being brutally honest here and I'm by no means they only one having VAS issues.



I just did some more testing last night and sadly my OOM frequency has rapidly increased (2-3 minutes max flight time). My VAS usage would start out about 2.5GB, takeoff and (keeping a single VC view and looking straight ahead only), it would jump to 3.4GB in <500ms, and then OOM 500ms later. Something is definitely not right ... I don't understand how VAS usage could increase almost 1GB in <500ms? All the 3rd party products I've installed claim to be "fully P3DV2 compatible" (FTX Global, FTX Vector, FS Global 2010 FTX Edition, Carenado B1900, Oslo 2.0).



Moving the sliders to the left only increasing the duration before I OOM ... but an OOM is eventual and certain, even using default aircraft. If I re-install P3DV2 from scratch and don't use any add-ons, I've been able to do 2 hour flights, but not much more before an OOM. Don't take this the wrong way, it's a great product and it's teasing me with fantastic visuals and performance, but this VAS issue is a serious problem for me. And if the only way for me to proceed with P3DV2 is to never load any 3rd party add-ons, I can't see that as being something good for either LM or myself?



Cheers, Rob.

Rob Ainscough

J van E
Posts: 172
Joined: Mon Nov 14, 2011 8:20 am

Postby J van E » Thu Jan 09, 2014 1:23 pm

I have to agree with Rob. What will the future of P3D be if you can only add just one or two addons? I love what P3D gives me but I feel very limited to what I can do. I would love to get a new and up to date PC for P3D but what is the use if I can't enable more options than I use now if that only limits my flying time because of VAS problems? I tested a GTX780 a short while ago and while performance was better (not very much better btw compared to my GTX580 so I returned the GPU) I did notice RAM usage with a few higher settings (autogen, water, shadows) was very high from the start and near to VAS limits pretty quickly. So if I would buy the power hardware to run P3D with higher settings, I would still have to limit myself to medium settings! And that's all with one texture addon and one addon plane...

WBard
Posts: 1034
Joined: Mon Aug 16, 2010 7:23 pm

Postby WBard » Thu Jan 09, 2014 2:12 pm

As Beau mentioned above, we have actually done a lot with legacy memory management, but the new features consume memory. You can tone down your texture resolution if it's higher than 1024 and avoid a lot of OOMs that way, or find a tradeoff with reflections, shadows, and texture resolution size that keeps you within memory limits.



We are going to include a memory management UI interface at some point once we iron out the rest of the initial v2 bugs to make it easier for users to see where and how their available resources are getting used.

J van E
Posts: 172
Joined: Mon Nov 14, 2011 8:20 am

Postby J van E » Thu Jan 09, 2014 4:29 pm



Quote:

Quote from WBard on January 9, 2014, 14:12

As Beau mentioned above, we have actually done a lot with legacy memory management, but the new features consume memory. You can tone down your texture resolution if it's higher than 1024 and avoid a lot of OOMs that way, or find a tradeoff with reflections, shadows, and texture resolution size that keeps you within memory limits.



We are going to include a memory management UI interface at some point once we iron out the rest of the initial v2 bugs to make it easier for users to see where and how their available resources are getting used.




That memory management UI interface sounds like a great idea! That would make it a lot easier to figure out which settings you for instance would have to lower a little in order to raise the settings which are important to you. Nice!



Apart from that: pity we have to find tradeoffs... ;) but that's what you get with a 32 bit program. So... bring on 64 bit in v2.2! :P (Kidding here, I know it's not thay easy, but well, I guess it IS the future...!)

User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Thu Jan 09, 2014 4:33 pm

Hey Wes,



Sadly going from 2048 to 1024 textures doesn't make much difference on my system in terms of OOM. Also, I was told by a 3rd party developer, there are NO textures above 1024 in P3DV2 - is this accurate?



I'm certain a memory management UI interface may help some folks, but isn't that only going to help them turn down visual settings to avoid OOM? I sorta don't understand the move to Multi-GPU support (v2.1) which is only going to encourage people to turn UP visual settings (not down) which will greet them with OOMs. It's puzzling to me why offer these visual options given the 32bit address space and apparent known memory management problems in C++ and/or OS? There is no ability to scale ... adding more RAM, adding GPUs, faster CPUs ... none of this is going to allow us to use all those wonderful visual settings that P3DV2 provides. It's an unobtainable and even cruel tease so long as we live in a 32bit address space. ;)



Cheers, Rob.
Rob Ainscough

patmarkus
Posts: 45
Joined: Tue Dec 17, 2013 4:35 pm

Postby patmarkus » Thu Jan 09, 2014 6:50 pm

Hi Wes,



I am just curious about something and I think I must agree with Rob's assesment.



In FSX I am using the PMDG 777, in combination with a lot of add-on airports, REX, FTX Global and some other addons.



I can only complete a flight and land if I reduce the traffic slider considerably before landing to free up some of the VAS, otherwise I get an OOM before I land.



You said that the new features consume more memory, so this means even more VAS is in use before using heavy add ons.



Am I crazy or does all this mean that the new features are going to make it impossible to use add ons such as planes from PMDG?



I am thinking this way because I was obviously at the limit of VAS usage in FSX, so even now you have added new features, I would be unable to use them as there is no VAS room left?



Is this a correct assessment or is my thinking wrong here?



It just makes me wonder if it is worth to keep putting money in the development of a 32bit app for you guys, even though the limit has been reached already.



I am really curious how this is going to develop, as I feel that it is going to be the one issue that is going to make this sim succeed or fail.

User avatar
Rob Ainscough
Posts: 2517
Joined: Sat Jan 05, 2013 6:46 pm
Location: Oregon USA

Postby Rob Ainscough » Fri Jan 10, 2014 3:03 am



Quote:

Quote from WBard on January 9, 2014, 14:12

As Beau mentioned above, we have actually done a lot with legacy memory management, but the new features consume memory. You can tone down your texture resolution if it's higher than 1024 and avoid a lot of OOMs that way, or find a tradeoff with reflections, shadows, and texture resolution size that keeps you within memory limits.



We are going to include a memory management UI interface at some point once we iron out the rest of the initial v2 bugs to make it easier for users to see where and how their available resources are getting used.




Wes, Pat, Kosta, Others



I seem to have made good progress tonight in resolving (working around) VAS issues. The combination (both not just one) of these settings:



1. Autogen Vegetation Density = Dense (down from Very Dense)

2. Autogen Building Density = Dense (down from Very Dense)





Has freed up 1.3GB of VAS and I no longer get the sudden and rapid VAS changes that were causing OOMs.



Wes, it might be worth a quick look/investigation around what is actually happening internally when you go from Dense to Very Dense on those two settings ... I wouldn't expect such a drastic change in VAS usage from only two minor changes. The programmer in me tells me something unexpected is getting triggered by these settings.



Anyway, just wanted to pass along my findings for others with VAS/OOM issues -- give those two settings a try and see if it helps ... made a huge difference for me and the visual change wasn't too drastic. Here is a video I made AFTER those two adjustments (long flight in an area I know is problematic using FTX Global, FTX Vector, FS Global 2010 FTX Edition (mesh), Aerosoft Oslo 2.0, Aerosoft Twin Otter Extended)- should be available in 1440p:



http://www.youtube.com/watch?v=dTVdFdLwkQ0



Rob

Rob Ainscough


Return to “Prepar3D Client Application Questions”

Who is online

Users browsing this forum: No registered users and 66 guests