OpenSceneGraph Forum Forum Index OpenSceneGraph Forum
Official forum which mirrors the existent OSG mailing lists. Messages posted here are forwarded to the mailing list and vice versa.
 
   FAQFAQ    SearchSearch    MemberlistMemberlist    RulesRules    UsergroupsUsergroups    RegisterRegister 
 Mail2Forum SettingsMail2Forum Settings  ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
   AlbumAlbum  OpenSceneGraph IRC ChatOpenSceneGraph IRC Chat   SmartFeedSmartFeed 

DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted


 
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General
View previous topic :: View next topic  
Author Message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sat Mar 24, 2018 1:53 pm    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi,

I have the following scenario:

1) A scene is using a database pager to load data by a pseudo-loader via the network; the viewer runs single threaded, the database pager uses default parameters
2) When the database pager is still loading data via the network the viewer is closed.
3) Because of the closing of the viewer the database pager's destructor is called but it seems to be that the database threads are still running (I put some output at the end of the database pager's destructor and inside database pager's database threads and the database threads are still producing output).
Looking at the database pager's destructor the threads are not finished but will stop only when their current task has finished, I think.
4) The database thread will crash at
Code:
DatabasePager::FindCompileableGLObjectsVisitor stateToCompile(_pager, _pager->getMarkerObject());
inside DatabasePager::DatabaseThread::run().
5) The reason for the crash is that DatabaseThread uses a simple pointer to the database pager. But the database pager is gone and therefore the stored pointer ("_pager") became invalid!

Am I doing anything wrong or is this a bug in OSG?

I think that "_pager" should be at least an observer of the database pager pointer and not a bare pointer. Or the destructor of DatabasePager must invalidate all its references stored in DatabasePager::DatabaseThread.

Cheers,
Hartwig
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sat Mar 24, 2018 3:25 pm    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi Hartwig,

What version of the OSG are you using? What platform? What compiler?

Robert.

On 24 March 2018 at 13:53, Hartwig Wiesmann <> wrote:
Quote:
Hi,

I have the following scenario:

1) A scene is using a database pager to load data by a pseudo-loader via the network; the viewer runs single threaded, the database pager uses default parameters
2) When the database pager is still loading data via the network the viewer is closed.
3) Because of the closing of the viewer the database pager's destructor is called but it seems to be that the database threads are still running (I put some output at the end of the database pager's destructor and inside database pager's database threads and the database threads are still producing output).
Looking at the database pager's destructor the threads are not finished but will stop only when their current task has finished, I think.
4) The database thread will crash at

Code:
DatabasePager::FindCompileableGLObjectsVisitor stateToCompile(_pager, _pager->getMarkerObject());

inside DatabasePager::DatabaseThread::run().
5) The reason for the crash is that DatabaseThread uses a simple pointer to the database pager. But the database pager is gone and therefore the stored pointer ("_pager") became invalid!

Am I doing anything wrong or is this a bug in OSG?

I think that "_pager" should be at least an observer of the database pager pointer and not a bare pointer. Or the destructor of DatabasePager must invalidate all its references stored in DatabasePager::DatabaseThread.

Cheers,
Hartwig

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=73157#73157








------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sat Mar 24, 2018 3:53 pm    Post subject:
Reply with quote

Hi,

I am using

- OSG 3.5.10
- mac OS 10.13
- Clang 9.0

And _OPENTHREADS_ATOMIC_USE_MUTEX is equal to 1.

Cheers,
Hartwig
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sat Mar 24, 2018 5:06 pm    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi Hartwig,

On 24 March 2018 at 15:53, Hartwig Wiesmann <> wrote:
Quote:
I am using

- OSG 3.5.10
- mac OS 10.13
- Clang 9.0

I recall doing some investigation work into some OSX/Clang issues last
year, and we couldn't resolve the issues as it looked to be low level
bug in Clang's set up of atomics. This might be the same issue for
you.

Could you try gcc?

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sat Mar 24, 2018 9:30 pm    Post subject:
Reply with quote

Hi Robert,

I think that it has nothing to do with Clang or GCC. I did some further investigations and probably found the bug. In DatabasePager::requestNodeFile you find the following code (I have added the preprocessor directive CRASH, initially equal to 1):
Code:
    if (!_startThreadCalled)
    {
        OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_run_mutex);

        if (!_startThreadCalled)
        {
            _startThreadCalled = true;
            _done = false;
            OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

            if (_databaseThreads.empty())
            {
                setUpThreads(
                    osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
                    osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
            }

#if (CRASH)
               for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
                dt_itr != _databaseThreads.end();
                ++dt_itr)
            {
                (*dt_itr)->startThread();
            }
#endif


In my scenario _startThreadCalled is equal to false and _databaseThreads.empty() returns true. Before doing any setup _startThreadCalled is set to true. Then, the threads are set up. Inside the setupThread method addDatabaseThread is called which starts the added thread because _startThreadCalled is already true:
Code:
    unsigned int pos = _databaseThreads.size();

    DatabaseThread* thread = new DatabaseThread(this, mode,name);

    thread->setProcessorAffinity(_affinity);

    _databaseThreads.push_back(thread);

    if (_startThreadCalled)
    {
        OSG_INFO<<"DatabasePager::startThread()"<<std::endl;
        thread->startThread();
    }


Afterwards, the threads are started again (see first code snippet):
Code:
for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
                dt_itr != _databaseThreads.end();
                ++dt_itr)
            {
                (*dt_itr)->startThread();
            }


If I set CRASH to zero the last code is not executed and there is no crash! So, it seems to be that all threads are created and started twice while they are registered and stopped only once!

Now, after the DatabasePager's destructor is called there are no threads anymore that produce any output!

Cheers,
Hartwig
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sun Mar 25, 2018 8:11 am    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

HI Hartwig,

Could you explain what you think the bug is as copying pasting some
bit of code without any explanation is a bit too obscure to divine
what you mean. I presume it's you who's added the #if 0 #endif around
the startThread() calls as this code is not the same as master/OSG-3.6

Robert.

On 24 March 2018 at 21:31, Hartwig Wiesmann <> wrote:
Quote:
Hi Robert,

I think that it has nothing to do with Clang or GCC. I did some further investigations and probably found the bug. In DatabasePager::requestNodeFile you find the following code:

Code:
if (!_startThreadCalled)
{
OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_run_mutex);

if (!_startThreadCalled)
{
_startThreadCalled = true;
_done = false;
OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

if (_databaseThreads.empty())
{
setUpThreads(
osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
}

#if 0
for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
dt_itr != _databaseThreads.end();
++dt_itr)
{
(*dt_itr)->startThread();
}
#endif




Thank you!

Cheers,
Hartwig

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=73162#73162








------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sun Mar 25, 2018 9:28 am    Post subject:
Reply with quote

Hi Robert,

sorry, that it was not clear enough. What I tried to explain is in summary that the database pager starts twice as many threads as it should but keeps track of only half of them. Therefore, when the viewer is closed the database pager only cancels half of them while the other half is still running. And the still running threads then access via the "_pager" pointer the non-existing database pager.

This is what happens in details and what I tried to explain by pasting the relevant code previously:
The database pager keeps track of the running threads in "_databaseThreads". When setting up "_databaseThreads" (currently empty) in requestNodeFile the threads are immediately created and started (in "setUpThreads"). To be precise, the threads are actually started in Thread::start() which is called from "setUpThreads".
At this moment N threads have been created, started and are tracked in "_databaseThreads".
Afterwards, the mentioned for-loop starts again(!) N threads but uses the same list (namely "_databaseThreads") to track them. Therefore, from the now running 2*N threads N are running untracked.

This can also be seen when looking with a debugger at the number of threads the database pager has been generated.

This bug can be solved by

a) moving the "_startThreadCalled" assignment further down
Code:
            _done = false;
            OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

            if (_databaseThreads.empty())
            {
                setUpThreads(
                    osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
                    osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
            }

               for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
                dt_itr != _databaseThreads.end();
                ++dt_itr)
            {
                (*dt_itr)->startThread();
            }
            _startThreadCalled = true;


or

b) inserting an else-statement to start either only newly added or the already existing threads

Code:
            _startThreadCalled = true;
            _done = false;
            OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

            if (_databaseThreads.empty())
            {
                setUpThreads(
                    osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
                    osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
            }
               else
               for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
                dt_itr != _databaseThreads.end();
                ++dt_itr)
            {
                (*dt_itr)->startThread();
            }


Which solution is chosen is a matter of taste.

I also recommend to document the functionality of "_startThreadCalled" because the wrong usage (or better hidden effects) of this flag in the above mentioned code causes the bug.

I can provide a patch but I first would like to make sure that my analysis are correct (I think so but I am not an expert).

Cheers,
Hartwig
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sun Mar 25, 2018 9:42 am    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi Hartwig,

Thanks for the explanation. I will need to a debugging to the OSG to
see the issue first hand to fully understand the issue.

Is it possible to recreate the 2*N threads being created with a
standard OSG example like osgviewer?

Robert.

On 25 March 2018 at 10:28, Hartwig Wiesmann <> wrote:
Quote:
Hi Robert,

sorry, that it was not clear enough. What I tried to explain is in summary that the database pager starts twice as many threads as it should but keeps track of only half of them. Therefore, when the viewer is closed the database pager only cancels half of them while the other half is still running. And the still running threads then access via the "_pager" pointer the non-existing database pager.

This is what happens in details and what I tried to explain by pasting the relevant code previously:
The database pager keeps track of the running threads in "_databaseThreads". When setting up "_databaseThreads" (currently empty) in requestNodeFile the threads are immediately created and started (in "setUpThreads"). To be precise, the threads are actually started in Thread::start() which is called from "setUpThreads".
At this moment N threads have been created, started and are tracked in "_databaseThreads".
Afterwards, the mentioned for-loop starts again(!) N threads but uses the same list (namely "_databaseThreads") to track them. Therefore, from the now running 2*N threads N are running untracked.

This can also be seen when looking with a debugger at the number of threads the database pager has been generated.

This bug can be solved by

a) moving the "_startThreadCalled" assignment further down

Code:
_done = false;
OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

if (_databaseThreads.empty())
{
setUpThreads(
osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
}

for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
dt_itr != _databaseThreads.end();
++dt_itr)
{
(*dt_itr)->startThread();
}
_startThreadCalled = true;




or

b) inserting an else-statement to start either only newly added or the already existing threads


Code:
_startThreadCalled = true;
_done = false;
OSG_INFO<<"DatabasePager::startThread()"<<std::endl;

if (_databaseThreads.empty())
{
setUpThreads(
osg::DisplaySettings::instance()->getNumOfDatabaseThreadsHint(),
osg::DisplaySettings::instance()->getNumOfHttpDatabaseThreadsHint());
}
else
for(DatabaseThreadList::const_iterator dt_itr = _databaseThreads.begin();
dt_itr != _databaseThreads.end();
++dt_itr)
{
(*dt_itr)->startThread();
}




Which solution is chosen is a matter of taste.

I also recommend to document the functionality of "_startThreadCalled" because the wrong usage (or better hidden effects) of this flag in the above mentioned code causes the bug.

I can provide a patch but I first would like to make sure that my analysis are correct (I think so but I am not an expert).

Cheers,
Hartwig

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=73164#73164








------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sun Mar 25, 2018 10:19 am    Post subject:
Reply with quote

Hi Robert,

I will try. My problem is that in my test project the database pager seems to be (time wise) initialised differently than in my real project.

Let's see.

Cheers,
Hartwig
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sun Mar 25, 2018 10:34 am    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi Hartwig,

Quote:
Is it possible to recreate the 2*N threads being created with a
standard OSG example like osgviewer?

I have added some debug output to DatabasePager and OpenThreads and
confirmed that some DatabaseThread are run twice.

I think there are two bugs here - first of DatabasePager.cpp is
calling startThead twice, secondly OpenThreads itself isn't catching
this.

OpenThreads::Thread::startThread() needs to be modified so that it
doesn't start a thread that has already been started. And
DatabasePager needs to fixed so it doesn't start a thread twice. Both
fixes are needed. I will now have a think about what is the cleanest
way to implement these fixes.

Cheers,
Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 11971

PostPosted: Sun Mar 25, 2018 11:47 am    Post subject:
DatabasePager::DatabaseThread is crashing when DatabasePager has been deleted
Reply with quote

Hi Hartwig et. al,

On 25 March 2018 at 11:29, Robert Osfield <> wrote:
Quote:
I think there are two bugs here - first of DatabasePager.cpp is
calling startThead twice, secondly OpenThreads itself isn't catching
this.

OpenThreads::Thread::startThread() needs to be modified so that it
doesn't start a thread that has already been started. And
DatabasePager needs to fixed so it doesn't start a thread twice. Both
fixes are needed. I will now have a think about what is the cleanest
way to implement these fixes.

I have now checked in fixes for OpenThreads and DatabasePager, these
are merged with master, OpenSceneGraph-3.4 and OpenSceneGraph-3.6
branches.

Changes are:
https://github.com/openscenegraph/OpenSceneGraph/commit/1224f400119e1e822bd83ef9a3f966bd8f90aaf9
https://github.com/openscenegraph/OpenSceneGraph/commit/aa69f509337269a60ea3f22711a2190e733a84d8

Could you test this out with your applications and let me know how you get on.

If it looks good I'll create a 3.6.0-rc2 to wrap them up.

Thanks for pinpointing the bug,
Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
hartwigw
Appreciator


Joined: 20 Mar 2009
Posts: 128

PostPosted: Sun Mar 25, 2018 3:11 pm    Post subject:
Reply with quote

Hi Robert,

I tested the modifications (using mac OS, modified OSG 3.5.10) and did not have any crashes anymore.

Thank you!

Cheers,
Hartwig

BTW: Pretty old bug, surprisingly nobody else had problems so far. Surprised I checked that the bug was probably introduced in 2008!
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Similar Topics
Topic Author Forum Replies Posted
No new posts get some "std::bad_alloc" e... surveyors General 3 Tue Apr 24, 2018 10:56 am View latest post
No new posts get std::bad_alloc exception in Datab... surveyors General 0 Mon Apr 02, 2018 4:30 am View latest post
No new posts DatabasePager insights Trajce Nikolov NICK General 1 Tue Feb 13, 2018 4:15 am View latest post
No new posts Crash in ReaderWriterCURL::readFile w... hartwigw General 3 Sun Dec 17, 2017 11:46 pm View latest post
No new posts Avoid call of SetupThreads in Databas... remoe Submission 3 Sun Apr 30, 2017 2:05 pm View latest post


Board Security Anti Bot Question MOD - phpBB MOD against Spam Bots
Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP