OpenSceneGraph Forum Forum Index OpenSceneGraph Forum
Official forum which mirrors the existent OSG mailing lists. Messages posted here are forwarded to the mailing list and vice versa.
 
   FAQFAQ    SearchSearch    MemberlistMemberlist    RulesRules    UsergroupsUsergroups    RegisterRegister 
 Mail2Forum SettingsMail2Forum Settings  ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
   AlbumAlbum  OpenSceneGraph IRC ChatOpenSceneGraph IRC Chat   SmartFeedSmartFeed 

Slow optimization and OpenFlight


 
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General
View previous topic :: View next topic  
Author Message
Andreas Ekstrand
Guest





PostPosted: Fri Nov 17, 2017 4:16 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi,

I'm still struggling with migrating from old OSG 3.2.1 to the latest developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an OpenFlight model with a large number of polygons, the optimization carried out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the culprit is MergeGeometryVisitor.

Here is an example with 150 000 polygons, it takes about 12s to load in osgviewer 3.2.1 here and what feels like forever to load in 3.5.8:
https://www.dropbox.com/s/lmapv0iatn20d1q/house_150k.flt

Has anyone seen this before? Any ideas how I can get it to behave like before?

Regards,
Andreas

------------------
Post generated by Mail2Forum
Back to top
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Fri Nov 17, 2017 4:37 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Andreas,

Did you try OSG-3.4.x? I'm wondering about when the regression in
performance might have happened.

I have downloaded the file and will have a look.

Robert.

On 17 November 2017 at 16:14, Andreas Ekstrand
<> wrote:
Quote:
Hi,

I'm still struggling with migrating from old OSG 3.2.1 to the latest
developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an
OpenFlight model with a large number of polygons, the optimization carried
out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the
culprit is MergeGeometryVisitor.

Here is an example with 150 000 polygons, it takes about 12s to load in
osgviewer 3.2.1 here and what feels like forever to load in 3.5.8:
https://www.dropbox.com/s/lmapv0iatn20d1q/house_150k.flt

Has anyone seen this before? Any ideas how I can get it to behave like
before?

Regards,
Andreas






------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Andreas Ekstrand
Guest





PostPosted: Sun Nov 19, 2017 2:14 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Robert,

Yes, I tried it now and OSG 3.4 loads the model in the same time as 3.2.1 did. I found a few versions lying around here and concluded that something must have happened with the optimization between 3.5.1 where it works as before and 3.5.6 where it stalls.

I'll keep digging but please let me know if you find something out.

Regards,
Andreas



On 2017-11-17 17:35, Robert Osfield wrote:

Quote:
Quote:
Hi Andreas,

Did you try OSG-3.4.x? I'm wondering about when the regression in
performance might have happened.

I have downloaded the file and will have a look.

Robert.

On 17 November 2017 at 16:14, Andreas Ekstrand
<> (
Only registered users can see emails on this board!
Get registred or enter the forums!
) wrote:
Quote:
Hi,

I'm still struggling with migrating from old OSG 3.2.1 to the latest
developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an
OpenFlight model with a large number of polygons, the optimization carried
out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the
culprit is MergeGeometryVisitor.

Here is an example with 150 000 polygons, it takes about 12s to load in
osgviewer 3.2.1 here and what feels like forever to load in 3.5.8:
https://www.dropbox.com/s/lmapv0iatn20d1q/house_150k.flt

Has anyone seen this before? Any ideas how I can get it to behave like
before?

Regards,
Andreas


_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org



------------------
Post generated by Mail2Forum
Back to top
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Sun Nov 19, 2017 3:30 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Andreas Ekstrand
Guest





PostPosted: Sun Nov 19, 2017 4:15 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Robert,

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

3.5.8: TOTAL: 121.4s
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator++ (78.5s)
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator-- (27.4s)
3.5.1: TOTAL: 12.7s
  • Registry::read -> Optimizer::optimize -> Optimizer::StateVisitor::optimize -> Node::setStateSet -> StateSet::~StateSet -> StateSet::clear -> StateAttribute::removeParent (2.4s)
  • Registry::read -> Optimizer::optimize -> Optimizer::MergeGeodesVisitor::mergeGeodes -> Node::~Node -> Node::setStateSet -> StateAttribute::removeParent (2.2s)
The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas



On 2017-11-19 16:28, Robert Osfield wrote:

Quote:
Quote:
HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org



------------------
Post generated by Mail2Forum
Back to top
Andreas Ekstrand
Guest





PostPosted: Fri Nov 24, 2017 1:12 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi,

Some more results from my investigations:

Back in February, the MergeGeometryVisitor was changed to be applied on Group instead of Geode by Jannik Heller (in CC). This has resulted in substantially longer optimization time of my loaded OpenFlight files with a large number of polygons. I assume it's a general performance problem with many geodes/geometries.

The removeChild call from MergeGeometryVisitor ::mergeGroup is the main culprit and takes about 87% of the time:
group.removeChild(*ditr);

Does Robert or Jannik have any idea about this? I don't feel comfortable enough with the code to make any changes, but this prevents me from upgrading my software to the latest OSG version, while I can't go back either since I need other fixes in 3.5.8. So I'm a bit stuck and would appreciate any help!

Regards,
Andreas



On 2017-11-19 17:13, Andreas Ekstrand wrote:

Quote:
Hi Robert,

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

3.5.8: TOTAL: 121.4s
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator++ (78.5s)
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator-- (27.4s)
3.5.1: TOTAL: 12.7s
  • Registry::read -> Optimizer::optimize -> Optimizer::StateVisitor::optimize -> Node::setStateSet -> StateSet::~StateSet -> StateSet::clear -> StateAttribute::removeParent (2.4s)
  • Registry::read -> Optimizer::optimize -> Optimizer::MergeGeodesVisitor::mergeGeodes -> Node::~Node -> Node::setStateSet -> StateAttribute::removeParent (2.2s)
The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas



On 2017-11-19 16:28, Robert Osfield wrote:

Quote:
Quote:
HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org




Quote:
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


------------------
Post generated by Mail2Forum
Back to top
Andreas Ekstrand
Guest





PostPosted: Fri Nov 24, 2017 1:46 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Forgot to mention that the geometry before and after optimization is identical in the different OSG versions, so the optimizer accomplishes the same thing, it just takes about 10 times longer.
/Andreas



On 2017-11-24 14:10, Andreas Ekstrand wrote:

Quote:
Hi,

Some more results from my investigations:

Back in February, the MergeGeometryVisitor was changed to be applied on Group instead of Geode by Jannik Heller (in CC). This has resulted in substantially longer optimization time of my loaded OpenFlight files with a large number of polygons. I assume it's a general performance problem with many geodes/geometries.

The removeChild call from MergeGeometryVisitor ::mergeGroup is the main culprit and takes about 87% of the time:
group.removeChild(*ditr);

Does Robert or Jannik have any idea about this? I don't feel comfortable enough with the code to make any changes, but this prevents me from upgrading my software to the latest OSG version, while I can't go back either since I need other fixes in 3.5.8. So I'm a bit stuck and would appreciate any help!

Regards,
Andreas



On 2017-11-19 17:13, Andreas Ekstrand wrote:

Quote:
Hi Robert,

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

3.5.8: TOTAL: 121.4s
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator++ (78.5s)
  • Registry::read -> Optimizer::optimize -> MergeGeometryVisitor::mergeGroup -> Group::removeChildren -> OpenThreads::Atomic::operator-- (27.4s)
3.5.1: TOTAL: 12.7s
  • Registry::read -> Optimizer::optimize -> Optimizer::StateVisitor::optimize -> Node::setStateSet -> StateSet::~StateSet -> StateSet::clear -> StateAttribute::removeParent (2.4s)
  • Registry::read -> Optimizer::optimize -> Optimizer::MergeGeodesVisitor::mergeGeodes -> Node::~Node -> Node::setStateSet -> StateAttribute::removeParent (2.2s)
The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas



On 2017-11-19 16:28, Robert Osfield wrote:

Quote:
Quote:
HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org




Quote:
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org



Quote:
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


------------------
Post generated by Mail2Forum
Back to top
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Mon Nov 27, 2017 12:58 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Andreas,

On 24 November 2017 at 13:44, Andreas Ekstrand
<> wrote:
Quote:
Forgot to mention that the geometry before and after optimization is
identical in the different OSG versions, so the optimizer accomplishes the
same thing, it just takes about 10 times longer.

I am currently investigating the issue, currently waiting on valgrind
--tool callgrind to complete to see what clues gives me. It's already
been running quite a while so am getting on with other avenues of
investigation while I wait.

The changes to Optimizer.cpp that you sent to osg-submissions suggests
that it's the removeChild() that is the performance bottleneck. The
changes you've proposed don't handle the case where a group can have
non Drawable children so can't be merged as is. It's still a
possibility of merging it with mods to address this.

I'd like to get down to route cause of the bottleneck though, my best
guess right now is it's an issue of parts of the implementation
triggering a O(N^2) behaviour, i.e. large number of children being
merged. It might be this can be resolved with the approach you took,
or it might be best adding better removeChildren() support that avoids
the O(N^2) behaviour, such as passing a list of nodes to remove.

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Mon Nov 27, 2017 1:29 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

HI Andreas,

I gave up the valgrind tool=callgrind because it was taking too long
to complete. What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

To get some context to why it's a bottlenck I put in some stats output
from the MergeGeometryVisitor::mergeGroup(..) and got the following
output:

$ osgconv house_150k.flt test3.osgb
Optimizer::MergeGeometryVisitor::mergeGroup() 148976
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=2324
Data written to 'test3.osgb'.

What this tells us that this dataset generated by the OpenFlight
loader is truly dreadful. It's just appalling how it's handling the
house_150k.flt database. I don't know if this can be traced back to
what the OSG's OpenFlight plugin is doing or whether the tool that
generated the .flt file is just has a completely awful export tool.
What I can say is that the fact that the OSG can handle at all such a
brain dead messed up dataset is a miracle.

The best way to resolve this issue is to refactor the
MergeGeometryVisitor so it handles these really bad datasets with more
grace, avoiding the pitfalls of an O(N^2) algorithm.

Having the core OSG "fix" really bad data is not a proper resolution.
it doesn't fix the fact the the OpenFlight loader is generating such a
dreadful scene graph. It might be that the OpenFlight loader is just
doing what the source database is tell it to do and isn't to blame,
but like the change MergeGeometryVisitor, it may be worth catching
this dreadful datacase on load and never generating the broken scene
graph. It's better to fix problems like this early rather than let it
get all the way to the osgUtil::Optimizer to patch fix things up, the
early it is done the less memory will be used and less memory will be
fragmented and better the performance will be.

I am afraid I just don't have the time to go chasing fixes to awful
3rd party data, I already have enough core OSG issues to do that
unpaid without taking on more. I will resolve the
MergeGeometryVisitor O(N^2) issue but improving the OpenFlight loader
to handle dreadful 3rd party data better I'll have to defer to the
community.

Cheers,
Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Andreas Ekstrand
Guest





PostPosted: Mon Nov 27, 2017 3:59 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Robert,

Thanks for looking into this!

Yes, you can argue that the dataset is awful, but it's actually just storing 150k triangles on one graph level, as separate OpenFlight face records, which is what the OpenFlight file format offers if you don't use its specific mesh nodes that aren't supported by all tools and don't offer full control of each triangle. OpenFlight is a modeling file format which is surely not visualization-friendly. I think the OpenFlight plugin simply uses the Optimizer to convert this to a dataset for effective visualization instead of converting to a more optimal scene graph to start with - a choice that I can't really say if it's good or bad, but the Optimizer should of course handle bad data in the best way possible.

The main issue for me though, is that OSG handles this dataset in a much slower way than before, hence my re-introduction of the previous optimization. But you're completely right, and I realize now, that other nodes than drawables aren't handle correctly. I haven't gotten used to drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and then uses the Optimizer itself to fix this is probably a separate and larger problem - if it indeed is a problem. But I believe it's necessary to at least get MergeGeometryVisitor to behave as fast as before, either by modifying my fixes or by rewriting parts of it. I hope you will find the time to decide on this soon, and please let me know if I can help!

Regards,
Andreas


On 2017-11-27 14:27, Robert Osfield wrote:

Quote:
Quote:
HI Andreas,

I gave up the valgrind tool=callgrind because it was taking too long
to complete. What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

To get some context to why it's a bottlenck I put in some stats output
from the MergeGeometryVisitor::mergeGroup(..) and got the following
output:

$ osgconv house_150k.flt test3.osgb
Optimizer::MergeGeometryVisitor::mergeGroup() 148976
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=3333
duplicateList.size()=2324
Data written to 'test3.osgb'.

What this tells us that this dataset generated by the OpenFlight
loader is truly dreadful. It's just appalling how it's handling the
house_150k.flt database. I don't know if this can be traced back to
what the OSG's OpenFlight plugin is doing or whether the tool that
generated the .flt file is just has a completely awful export tool.
What I can say is that the fact that the OSG can handle at all such a
brain dead messed up dataset is a miracle.

The best way to resolve this issue is to refactor the
MergeGeometryVisitor so it handles these really bad datasets with more
grace, avoiding the pitfalls of an O(N^2) algorithm.

Having the core OSG "fix" really bad data is not a proper resolution.
it doesn't fix the fact the the OpenFlight loader is generating such a
dreadful scene graph. It might be that the OpenFlight loader is just
doing what the source database is tell it to do and isn't to blame,
but like the change MergeGeometryVisitor, it may be worth catching
this dreadful datacase on load and never generating the broken scene
graph. It's better to fix problems like this early rather than let it
get all the way to the osgUtil::Optimizer to patch fix things up, the
early it is done the less memory will be used and less memory will be
fragmented and better the performance will be.

I am afraid I just don't have the time to go chasing fixes to awful
3rd party data, I already have enough core OSG issues to do that
unpaid without taking on more. I will resolve the
MergeGeometryVisitor O(N^2) issue but improving the OpenFlight loader
to handle dreadful 3rd party data better I'll have to defer to the
community.

Cheers,
Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org



------------------
Post generated by Mail2Forum
Back to top
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Mon Nov 27, 2017 4:27 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Hi Andreas,

I have checked in a refactor of MergeGeometry visitor that makes it
work properly with groups containing any type of nodes, and avoids the
O(N^2) removChild() usage. In the end I took the line of least
resistance and just used the approach of removing all children and
adding back the ndoes.

On 27 November 2017 at 15:56, Andreas Ekstrand
<> wrote:
Quote:
Yes, you can argue that the dataset is awful, but it's actually just storing
150k triangles on one graph level, as separate OpenFlight face records,
which is what the OpenFlight file format offers if you don't use its
specific mesh nodes that aren't supported by all tools and don't offer full
control of each triangle. OpenFlight is a modeling file format which is
surely not visualization-friendly. I think the OpenFlight plugin simply uses
the Optimizer to convert this to a dataset for effective visualization
instead of converting to a more optimal scene graph to start with - a choice
that I can't really say if it's good or bad, but the Optimizer should of
course handle bad data in the best way possible.

No need to argue whether it's awful or not, there isn't any debate
when it comes to real-time performance : this particular database is
stored in the worst possible way for memory utilization and
performance

Use of the osgUtil::Optimizer is a very crude fix for bad databases.
It's far better to just create a scene graph in an efficient form
right from the start. Optimizing bad databases at best leads to
fragment memory, it really is just papering over cracks and is
unlikely to ever produce optimal databases.


Quote:
The main issue for me though, is that OSG handles this dataset in a much
slower way than before, hence my re-introduction of the previous
optimization. But you're completely right, and I realize now, that other
nodes than drawables aren't handle correctly. I haven't gotten used to
drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and
then uses the Optimizer itself to fix this is probably a separate and larger
problem - if it indeed is a problem.

Yep, big topic. I'm familiar with the problems that the OpenFlight
loader generates, but less so with Creator or other tools that
generate the content. I'd much rather the OpenFlight loader builds
more efficient database to start with rather than rely upon the
Optimizer to fix problems once the data is loaded.

Quote:
But I believe it's necessary to at
least get MergeGeometryVisitor to behave as fast as before, either by
modifying my fixes or by rewriting parts of it. I hope you will find the
time to decide on this soon, and please let me know if I can help!

The fix I have checked in master:

https://github.com/openscenegraph/OpenSceneGraph/commit/bc4a9d9dd0158e1e99d0ca1d1d95ef204220a560

With performance testing of just the merge geometry on your house
dataset I saw a 70sec merge geometry before, and 0.1 sec merge after
my changes.

I've got plenty of other stuff to get on with so I'll assume this
issue is now resolved.

Cheers,
Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Andreas Ekstrand
Guest





PostPosted: Mon Nov 27, 2017 4:45 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Thanks Robert! Works fine here as well, and I think the line of least resistance was the right way this time.

Any plans on a 3.5.9 release soon? I'm now facing the problem of either waiting until the next release, or basing my software on 3.5.8 with this modification and thus implicitly recommending users to use 3.5.8 to be compatible but with the implication that their OpenFlight models might be load slowly...

/Andreas



On 2017-11-27 17:25, Robert Osfield wrote:

Quote:
Quote:
Hi Andreas,

I have checked in a refactor of MergeGeometry visitor that makes it
work properly with groups containing any type of nodes, and avoids the
O(N^2) removChild() usage. In the end I took the line of least
resistance and just used the approach of removing all children and
adding back the ndoes.

On 27 November 2017 at 15:56, Andreas Ekstrand
<> (
Only registered users can see emails on this board!
Get registred or enter the forums!
) wrote:
Quote:
Yes, you can argue that the dataset is awful, but it's actually just storing
150k triangles on one graph level, as separate OpenFlight face records,
which is what the OpenFlight file format offers if you don't use its
specific mesh nodes that aren't supported by all tools and don't offer full
control of each triangle. OpenFlight is a modeling file format which is
surely not visualization-friendly. I think the OpenFlight plugin simply uses
the Optimizer to convert this to a dataset for effective visualization
instead of converting to a more optimal scene graph to start with - a choice
that I can't really say if it's good or bad, but the Optimizer should of
course handle bad data in the best way possible.
No need to argue whether it's awful or not, there isn't any debate
when it comes to real-time performance : this particular database is
stored in the worst possible way for memory utilization and
performance

Use of the osgUtil::Optimizer is a very crude fix for bad databases.
It's far better to just create a scene graph in an efficient form
right from the start. Optimizing bad databases at best leads to
fragment memory, it really is just papering over cracks and is
unlikely to ever produce optimal databases.


Quote:
The main issue for me though, is that OSG handles this dataset in a much
slower way than before, hence my re-introduction of the previous
optimization. But you're completely right, and I realize now, that other
nodes than drawables aren't handle correctly. I haven't gotten used to
drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and
then uses the Optimizer itself to fix this is probably a separate and larger
problem - if it indeed is a problem.
Yep, big topic. I'm familiar with the problems that the OpenFlight
loader generates, but less so with Creator or other tools that
generate the content. I'd much rather the OpenFlight loader builds
more efficient database to start with rather than rely upon the
Optimizer to fix problems once the data is loaded.

Quote:
But I believe it's necessary to at
least get MergeGeometryVisitor to behave as fast as before, either by
modifying my fixes or by rewriting parts of it. I hope you will find the
time to decide on this soon, and please let me know if I can help!
The fix I have checked in master:

https://github.com/openscenegraph/OpenSceneGraph/commit/bc4a9d9dd0158e1e99d0ca1d1d95ef204220a560

With performance testing of just the merge geometry on your house
dataset I saw a 70sec merge geometry before, and 0.1 sec merge after
my changes.

I've got plenty of other stuff to get on with so I'll assume this
issue is now resolved.

Cheers,
Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org



------------------
Post generated by Mail2Forum
Back to top
Alberto Luaces
Guest





PostPosted: Mon Nov 27, 2017 4:48 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Robert Osfield writes:

Quote:
I gave up the valgrind tool=callgrind because it was taking too long
to complete. What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

For these matters the "perf" utility in the linux-perf package reaches
near-uninstrumented performance:

$ perf record <exe_and_arguments> # Gather results

$ perf report # Inspect results

Depending on the system configuration, running previously a

$ echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid

might be required.

--
Alberto



------------------
Post generated by Mail2Forum
Back to top
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12027

PostPosted: Mon Nov 27, 2017 4:49 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Thanks for the tip Alberto.

On 27 November 2017 at 16:46, Alberto Luaces <> wrote:
Quote:
Robert Osfield writes:

Quote:
I gave up the valgrind tool=callgrind because it was taking too long
to complete. What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

For these matters the "perf" utility in the linux-perf package reaches
near-uninstrumented performance:

$ perf record <exe_and_arguments> # Gather results

$ perf report # Inspect results

Depending on the system configuration, running previously a

$ echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid

might be required.

--
Alberto




------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Andreas Ekstrand
Guest





PostPosted: Mon Nov 27, 2017 9:41 pm    Post subject:
Slow optimization and OpenFlight
Reply with quote

Sounds great, Robert! The latest master builds fine here.
/Andreas



On 2017-11-27 17:47, Robert Osfield wrote:

Quote:
Quote:
On 27 November 2017 at 16:39, Andreas Ekstrand
<> (
Only registered users can see emails on this board!
Get registred or enter the forums!
) wrote:
Quote:
Thanks Robert! Works fine here as well, and I think the line of least
resistance was the right way this time.

Any plans on a 3.5.9 release soon? I'm now facing the problem of either
waiting until the next release, or basing my software on 3.5.8 with this
modification and thus implicitly recommending users to use 3.5.8 to be
compatible but with the implication that their OpenFlight models might be
load slowly...

I was planning to tag 3.5.9 in the next day or so. There isn't any
major feature I'm waiting on, so as long as it compiles and has no
regressions I can tag it right away,

Robert.



------------------
Post generated by Mail2Forum
Back to top
Display posts from previous:   
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Similar Topics
Topic Author Forum Replies Posted
No new posts Unbounded growth in OpenFlight Reader... Daniel Emminizer, Code... General 6 Tue Jul 31, 2018 9:55 am View latest post
No new posts Why is osgText::Text very slow when i... OmegaDoom General 1 Wed Jul 04, 2018 5:31 am View latest post
No new posts Exporting OpenFlight Models Daniel Emminizer, Code... General 2 Wed Jun 27, 2018 5:02 pm View latest post
No new posts Geometries disappearing at optimization Andreas Ekstrand General 3 Mon Apr 02, 2018 12:08 am View latest post
No new posts ViewerBase::frame() method slow after... gambr General 8 Fri Jul 21, 2017 9:11 am View latest post


Board Security Anti Bot Question MOD - phpBB MOD against Spam Bots
Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP