OpenSceneGraph Forum Forum Index OpenSceneGraph Forum
Official forum which mirrors the existent OSG mailing lists. Messages posted here are forwarded to the mailing list and vice versa.
 
   FAQFAQ    SearchSearch    MemberlistMemberlist    RulesRules    UsergroupsUsergroups    RegisterRegister 
 Mail2Forum SettingsMail2Forum Settings  ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
   AlbumAlbum  OpenSceneGraph IRC ChatOpenSceneGraph IRC Chat   SmartFeedSmartFeed 

[solvedish] VPBmaster consumes over 16GB of RAM


 
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> VirtualPlanetBuilder [vpb]
View previous topic :: View next topic  
Author Message
kingarthur10
Newbie


Joined: 17 Apr 2013
Posts: 13

PostPosted: Wed Apr 17, 2013 10:28 pm    Post subject:
[solvedish] VPBmaster consumes over 16GB of RAM
Reply with quote

Hello all,

I'm having some significant problems using VPB processing extremely large datasets. For this example, we'll take a look at the ASTER dataset which comprises 528.2GB of data split among 21,844 GeoTIFFs. The first problem I had was an error about too many open file handles. I upped the allowable number of open files from 4096 to 65536 through PAM and limits.conf. No longer do I receive errors pertaining to open file limit issues, but now I'm seeing vpbmaster consuming over 16GB of system RAM when trying to generate databases (16GB is the amount of available RAM I have on the system, so it typically starts eating into significant SWAP and slows the system to a crawl).

Is there a method to help limit vpbmaster from consuming that much RAM during normal use?


Question two:
I've attempted to process just one hemisphere at a time (North and South separately) to avoid issues with the out of memory, and it seemed to work great (only consumed about 4GB of RAM at any time). I pointed vpbmaster to a directory with only Northern hemisphere items, and the processing commenced normally. But, it began failing and blacklisting all of the cores as soon as it started processing southern hemisphere items. I was using the command
Quote:
vpbmaster --geocentric --terrain -d /path/to/northernhemisphere -O Compressor=zlib -o output.osgb

Anyway, I decided to look in the code, and I commented out the blacklist command in the MachinePool.cpp and uncommented the IGNORE_FAILED_TASK, and everything worked fine from that point forward. Is there a reason that blacklisting is the first thing VPB goes to? Am I doing something wrong by allowing it to just ignore the failed tile and move on?

Thanks so much, I'd love to hear your inputs!

Arthur


Last edited by kingarthur10 on Tue Jul 09, 2013 5:03 pm; edited 1 time in total
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 10892

PostPosted: Tue Apr 23, 2013 8:36 am    Post subject:
VPBmaster consumes over 16GB of RAM
Reply with quote

Hi Arthur,

I haven't worked with your specific data but in general vpbmaster should consume massive amounts of memory.  Could it be that the osgdem tasks that it spawns are the problem?  Are you running on a cluster of machines or on a single machine multi-threaded?


Robert.

------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
kingarthur10
Newbie


Joined: 17 Apr 2013
Posts: 13

PostPosted: Wed Apr 24, 2013 4:18 pm    Post subject:
Reply with quote

Hello Robert,

I have two different machines that I've got functioning independently of one another, one with 16GB of RAM and one with 32GB of RAM. I'm currently attempting to process the ASTER dataset on the 16GB of RAM machine (Quad Core i5).

For the dataset statistics I provided above, how much RAM do you think it should consume? Is there a way to dynamically add or remove cores depending on predicted RAM useage for a dataset? Or do you have to just monitor the system useage, kill VPB when you think you can add a second core to the system, and restart using the --tasks command?

Does the native clustering virtualize RAM? Or should I attempt to delve back into SGE or another resource distribution service?

Thanks for the reply, by the way. I appreciate it!
Back to top
View user's profile Send private message
kingarthur10
Newbie


Joined: 17 Apr 2013
Posts: 13

PostPosted: Tue Jul 09, 2013 5:06 pm    Post subject:
Reply with quote

So, I was finally able to process 1/4th of the dataset at a time with 32GB of RAM and a 230GB swap partition on an SSD.

Just wow. I had no idea that a high-resolution (~15mpp) full-earth DEM would need those kind of resources. I hope that I will be able to add-to the database rather than needing 4 independent databases of the DEM for the whole earth.

PS: I was running this through osgdem (VPB would always fail out...my guess is that I'd need much more RAM than the 260GB available to the system).

Thanks to all that discussed the issues,
Arthur
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> VirtualPlanetBuilder [vpb] All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Similar Topics
Topic Author Forum Replies Posted
No new posts vpbmaster appears to slow down after ... etunko VirtualPlanetBuilder [vpb] 3 Wed Aug 31, 2016 9:35 am View latest post
No new posts clip/mask with osgdem / vpbmaster maya leonard General 1 Tue Mar 31, 2015 12:02 pm View latest post
No new posts Unusual Error while running vpbmaster Informant VirtualPlanetBuilder [vpb] 11 Mon Nov 10, 2014 11:13 am View latest post
No new posts vpbmaster --splits ToninoTarsi VirtualPlanetBuilder [vpb] 2 Mon Apr 15, 2013 8:55 pm View latest post
No new posts why the ive created by osgDem more bu... robertosfield VirtualPlanetBuilder [vpb] 0 Wed Aug 22, 2012 4:25 pm View latest post


Board Security Anti Bot Question MOD - phpBB MOD against Spam Bots
Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP