OpenSceneGraph Forum Forum Index OpenSceneGraph Forum
Official forum which mirrors the existent OSG mailing lists. Messages posted here are forwarded to the mailing list and vice versa.
 
   FAQFAQ    SearchSearch    MemberlistMemberlist    RulesRules    UsergroupsUsergroups    RegisterRegister 
 Mail2Forum SettingsMail2Forum Settings  ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
   AlbumAlbum  OpenSceneGraph IRC ChatOpenSceneGraph IRC Chat   SmartFeedSmartFeed 

OpenThreads/pthreads & OpenMP on RHEL5.2

Goto page Previous  1, 2
 
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General
View previous topic :: View next topic  
Author Message
Robert Osfield
Guest





PostPosted: Wed Oct 15, 2008 5:07 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 6:01 PM, Csaba Halász <> wrote:
Quote:
Here is what stats I get: http://imagebin.ca/view/lDtnJ9ET.html
I have tried your debugging mods, with the osgcamera example I get the
expected printout:

numProcessors = 2
cpunum=1
cpunum=0
cpunum=1
running and setting thread 0xd91d18 for cpunum=0
running and setting thread 0xcb31c8 for cpunum=1
running and setting thread 0xd1e528 for cpunum=1

This looks correct.

Quote:
With our app (flightgear), however, I don't get any. I might have
messed up the compilation, but do you think there is another code path
to set threading model that could circumvent your prints?
In the meantime, I'll investigate too.

I don't know anything about flight gear code I'm afraid. There is
only one method in ViewerBase that sets up threading so it should be
called your method if flight gear is linking your app and set up
threading the normal way.

Robert.


------------------
Post generated by Mail2Forum
Back to top
Robert Osfield
Guest





PostPosted: Wed Oct 15, 2008 5:08 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 6:01 PM, Csaba Halász <> wrote:
Quote:
On Wed, Oct 15, 2008 at 6:47 PM, Robert Osfield
<> wrote:
Quote:

On Wed, Oct 15, 2008 at 5:30 PM, Csaba Halász <> wrote:

Quote:
As a related question, in the stats display if multiple cpus are used
properly, should that manifest itself as overlapping bars? Because I
don't see any such, the different phases (draw, cull) for the cameras
all appear to be run after each other. I am now building osg with your
debug modifications, to see if that tells me something.

On my quad core system I certain do see overlap. Milage will vary
across hardware, OS and OSG usage.

Here is what stats I get: http://imagebin.ca/view/lDtnJ9ET.html
I have tried your debugging mods, with the osgcamera example I get the
expected printout:

numProcessors = 2
cpunum=1
cpunum=0
cpunum=1
running and setting thread 0xd91d18 for cpunum=0
running and setting thread 0xcb31c8 for cpunum=1
running and setting thread 0xd1e528 for cpunum=1

With our app (flightgear), however, I don't get any. I might have
messed up the compilation, but do you think there is another code path
to set threading model that could circumvent your prints?
In the meantime, I'll investigate too.

--
Csaba




------------------
Post generated by Mail2Forum
Back to top
Csaba Halász
Guest





PostPosted: Wed Oct 15, 2008 5:23 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 7:07 PM, Robert Osfield
<> wrote:
Quote:
On Wed, Oct 15, 2008 at 6:01 PM, Csaba Halász <> wrote:
Quote:

Here is what stats I get: http://imagebin.ca/view/lDtnJ9ET.html
I have tried your debugging mods, with the osgcamera example I get the
expected printout:

With our app (flightgear), however, I don't get any. I might have
messed up the compilation, but do you think there is another code path
to set threading model that could circumvent your prints?
In the meantime, I'll investigate too.

I don't know anything about flight gear code I'm afraid. There is
only one method in ViewerBase that sets up threading so it should be
called your method if flight gear is linking your app and set up
threading the normal way.

As you can see on the screenshot, the statistics show that the
threading model is set, but no printout and the bars don't overlap
either.
The threading is set like so:
viewer = new osgViewer::Viewer;
viewer->setThreadingModel(osgViewer::Viewer::CullThreadPerCameraDrawThreadPerContext);

It seems to be loading the freshly compiled osg libs (fails to start
if I rename them)
We also use a databasepager, that seems to run on a different thread
as expected.

--
Csaba


------------------
Post generated by Mail2Forum
Back to top
Csaba Halász
Guest





PostPosted: Wed Oct 15, 2008 5:46 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 7:23 PM, Csaba Halász <> wrote:
Quote:

The threading is set like so:
viewer = new osgViewer::Viewer;
viewer->setThreadingModel(osgViewer::Viewer::CullThreadPerCameraDrawThreadPerContext);

And that's exactly the problem ... setting the threading mode before
the viewer is realized does not start threading.
I added an explicit call to startThreading() later, and get the printouts now Smile

--
Csaba


------------------
Post generated by Mail2Forum
Back to top
Robert Osfield
Guest





PostPosted: Wed Oct 15, 2008 6:28 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 6:45 PM, Csaba Halász <> wrote:
Quote:
On Wed, Oct 15, 2008 at 7:23 PM, Csaba Halász <> wrote:
Quote:

The threading is set like so:
viewer = new osgViewer::Viewer;
viewer->setThreadingModel(osgViewer::Viewer::CullThreadPerCameraDrawThreadPerContext);

And that's exactly the problem ... setting the threading mode before
the viewer is realized does not start threading.
I added an explicit call to startThreading() later, and get the printouts now Smile

This sounds like a bug in osgViewer, it shouldn't matter which order
your set the threading model.

Has the stats graph changed at all?

Robert.


------------------
Post generated by Mail2Forum
Back to top
Csaba Halász
Guest





PostPosted: Wed Oct 15, 2008 6:44 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 8:28 PM, Robert Osfield
<> wrote:
Quote:
On Wed, Oct 15, 2008 at 6:45 PM, Csaba Halász <> wrote:
Quote:
On Wed, Oct 15, 2008 at 7:23 PM, Csaba Halász <> wrote:
Quote:

The threading is set like so:
viewer = new osgViewer::Viewer;
viewer->setThreadingModel(osgViewer::Viewer::CullThreadPerCameraDrawThreadPerContext);

And that's exactly the problem ... setting the threading mode before
the viewer is realized does not start threading.
I added an explicit call to startThreading() later, and get the printouts now Smile

This sounds like a bug in osgViewer, it shouldn't matter which order
your set the threading model.

Hmm, upon closer look Viewer::realize() does call setUpThreading()
which in turn calls startThreading().
Gotta do some debugging why it doesn't work for me.

Quote:
Has the stats graph changed at all?

Yes, I now have overlapping bars. Unfortunately our code is doing
something from the wrong thread, so
CullThreadPerCameraDrawThreadPerContext doesn't work yet and even
CullDrawThreadPerContext crashes after some time. But these problems
are likely our fault.

--
Csaba


------------------
Post generated by Mail2Forum
Back to top
Ralph R. Peters
Guest





PostPosted: Wed Oct 15, 2008 6:59 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

Hi Robert & all,

I did the "simple thing" to test if the call pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) was causing OpenMP problems.
A code snippet from the function SetProcessorAffinityOfCurrentThread in
PThread.c++ follows. Changing "doit" from true to false and recompiling
lets me test this call.

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

#if defined(HAVE_PTHREAD_SETAFFINITY_NP)
std::cout << "SetProcessorAffinityOfCurrentThread: cpunum=" <<
cpunum << " sizeof(cpumask)=" << sizeof(cpumask) << std::endl;

bool doit = true;
if(doit)
{
pthread_setaffinity_np( pthread_self(), sizeof(cpumask),
&cpumask);
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask)
called immediately before this printout" << std::endl;
}
else
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask) NOT
called immediately before this printout result"
<< std::endl;
#elif defined(HAVE_THREE_PARAM_SCHED_SETAFFINITY)

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

The result was what I suspected. If the call to pthread_setaffinity_np
is NOT made then OpenMP recognizes that there are 8 processors,
otherwise it thinks that there is only 1. In both cases I used "strace
-f -e sched_setaffinity -e sched_getaffinity" to keep an eye on system
calls. Output for both cases is listed below.

I am just trying to use OpenMP inside an application that uses
OpenSceneGraph. What needs to be done to fix this problem, wherever it
may be, is "above my current paygrade". Smile

However, getting our application (umbra -
http://robotics.sandia.gov/UMBRA.html) to use OpenMP may be important to
many of us.

Thanks,
Ralph


************************************************************
Call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
sched_getaffinity(32493, 128, { ff, 0, 0, 0 }) = 32
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) called immediately before
this printout
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf
fileType: lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:1 MAX number of OpenMP
threads 1
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()
****************************************************************
do NOT call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) NOT called immediately before
this printout result
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf
fileType: lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:8 MAX number of OpenMP
threads 8
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()

************************************************************************



------------------
Post generated by Mail2Forum
Back to top
Robert Osfield
Guest





PostPosted: Wed Oct 15, 2008 7:25 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

Hi Ralf,

I'm afraid there isn't anything more I can do to help you, as I have
absolutely no experience with OpenMP, and am not an expert on the
pthread affinity functions. Perhaps others on the list might be able
to help.

What it does look like is that pthread_setaffinity_np/OpenMP
interoprability is buggy, and is beyond the scope of the
OSG/OpenThreads. If there are better pthread calls/settings to use to
avoid the OpenMP interoprability problems then I'm open to integrating
these, but I personally can't say what these might be or whether it
might be possible.

The only other thing that might be doable on the OSG side is to allow
you disable the affinity setting functions completely for your app.
This would break some of the performance benefits of the threading in
the viewer class though, so this would be far from an ideal solution.

Robert.

On Wed, Oct 15, 2008 at 7:58 PM, Ralph R. Peters <> wrote:
Quote:
Hi Robert & all,

I did the "simple thing" to test if the call pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) was causing OpenMP problems. A
code snippet from the function SetProcessorAffinityOfCurrentThread in
PThread.c++ follows. Changing "doit" from true to false and recompiling
lets me test this call.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

#if defined(HAVE_PTHREAD_SETAFFINITY_NP)
std::cout << "SetProcessorAffinityOfCurrentThread: cpunum=" << cpunum
<< " sizeof(cpumask)=" << sizeof(cpumask) << std::endl;

bool doit = true;
if(doit)
{
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask);
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask) called
immediately before this printout" << std::endl;
}
else
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask) NOT
called immediately before this printout result"
<< std::endl;
#elif defined(HAVE_THREE_PARAM_SCHED_SETAFFINITY)

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

The result was what I suspected. If the call to pthread_setaffinity_np is
NOT made then OpenMP recognizes that there are 8 processors, otherwise it
thinks that there is only 1. In both cases I used "strace -f -e
sched_setaffinity -e sched_getaffinity" to keep an eye on system calls.
Output for both cases is listed below.

I am just trying to use OpenMP inside an application that uses
OpenSceneGraph. What needs to be done to fix this problem, wherever it may
be, is "above my current paygrade". Smile

However, getting our application (umbra -
http://robotics.sandia.gov/UMBRA.html) to use OpenMP may be important to
many of us.

Thanks,
Ralph


************************************************************
Call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
sched_getaffinity(32493, 128, { ff, 0, 0, 0 }) = 32
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np( pthread_self(),
sizeof(cpumask), &cpumask) called immediately before this printout
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf fileType:
lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:1 MAX number of OpenMP
threads 1
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()
****************************************************************
do NOT call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np( pthread_self(),
sizeof(cpumask), &cpumask) NOT called immediately before this printout
result
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf fileType:
lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:8 MAX number of OpenMP
threads 8
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()

************************************************************************




------------------
Post generated by Mail2Forum
Back to top
Csaba Halász
Guest





PostPosted: Wed Oct 15, 2008 7:46 pm    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

On Wed, Oct 15, 2008 at 8:44 PM, Csaba Halász <> wrote:
Quote:
On Wed, Oct 15, 2008 at 8:28 PM, Robert Osfield
<> wrote:

Quote:
This sounds like a bug in osgViewer, it shouldn't matter which order
your set the threading model.

Hmm, upon closer look Viewer::realize() does call setUpThreading()
which in turn calls startThreading().
Gotta do some debugging why it doesn't work for me.

Got it:

int ViewerBase::run()
{
if (!isRealized())
{
realize();
}

But the isRealized() function doesn't really say if realize() has been
called or not, it just checks if *any* of the gc's are realized. Our
code happened to call realize on the gc, so the viewer's realize()
never got called. Depending on whether it is allowed to realize gc's
manually it is either a bug or not Smile

--
Csaba


------------------
Post generated by Mail2Forum
Back to top
J.P. Delport
Guest





PostPosted: Thu Oct 16, 2008 6:19 am    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

Hi Ralph,

if you can make a small OT vs OpenMP test app that people can run on a
variety of Linux flavours, maybe something will jump out. E.g. how
"standard" is the pthread lib in RHEL5.2?

jp

Ralph R. Peters wrote:
Quote:
Hi Robert & all,

I did the "simple thing" to test if the call pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) was causing OpenMP problems.
A code snippet from the function SetProcessorAffinityOfCurrentThread in
PThread.c++ follows. Changing "doit" from true to false and recompiling
lets me test this call.
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

#if defined(HAVE_PTHREAD_SETAFFINITY_NP)
std::cout << "SetProcessorAffinityOfCurrentThread: cpunum=" <<
cpunum << " sizeof(cpumask)=" << sizeof(cpumask) << std::endl;

bool doit = true;
if(doit)
{
pthread_setaffinity_np( pthread_self(), sizeof(cpumask),
&cpumask);
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask)
called immediately before this printout" << std::endl;
}
else
std::cout << "SetProcessorAffinityOfCurrentThread:
pthread_setaffinity_np( pthread_self(), sizeof(cpumask), &cpumask) NOT
called immediately before this printout result"
<< std::endl;
#elif defined(HAVE_THREE_PARAM_SCHED_SETAFFINITY)

&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&

The result was what I suspected. If the call to pthread_setaffinity_np
is NOT made then OpenMP recognizes that there are 8 processors,
otherwise it thinks that there is only 1. In both cases I used "strace
-f -e sched_setaffinity -e sched_getaffinity" to keep an eye on system
calls. Output for both cases is listed below.

I am just trying to use OpenMP inside an application that uses
OpenSceneGraph. What needs to be done to fix this problem, wherever it
may be, is "above my current paygrade". Smile

However, getting our application (umbra -
http://robotics.sandia.gov/UMBRA.html) to use OpenMP may be important to
many of us.

Thanks,
Ralph


************************************************************
Call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
sched_getaffinity(32493, 128, { ff, 0, 0, 0 }) = 32
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) called immediately before
this printout
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf
fileType: lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(32493, 128, { 1, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:1 MAX number of OpenMP
threads 1
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()
****************************************************************
do NOT call pthread_setaffinity_np:

camera number: 0
SetProcessorAffinityOfCurrentThread: cpunum=0 sizeof(cpumask)=128
SetProcessorAffinityOfCurrentThread: pthread_setaffinity_np(
pthread_self(), sizeof(cpumask), &cpumask) NOT called immediately before
this printout result
INFO> navigation mode set to umbra
Warning: font file "fonts/arial.ttf" not found.
Setting savConfigCB to '::guiWrapper saveConfig'
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
sLoadFile filename:/home/vision_data/geometry_objects/test.cdf
fileType: lineRowCnt=-1 lineColCnt=-1
std::string UmbModCDFDataSet::LoadFile(const s....

Enter test_openmp()
sched_getaffinity(1393, 128, { ff, 0, 0, 0 }) = 32
OPENMP is 200505 Number of processors available:8 MAX number of OpenMP
threads 8
GOMP_CPU_AFFINITY is unknown

Exit test_openmp()

************************************************************************




--
This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html.

This message has been scanned for viruses and dangerous content by MailScanner,
and is believed to be clean. MailScanner thanks Transtec Computers for their support.



------------------
Post generated by Mail2Forum
Back to top
Robert Osfield
Guest





PostPosted: Thu Oct 16, 2008 8:12 am    Post subject:
OpenThreads/pthreads & OpenMP on RHEL5.2
Reply with quote

Hi Csaba,

On Wed, Oct 15, 2008 at 8:46 PM, Csaba Halász <> wrote:
Quote:
Got it:

int ViewerBase::run()
{
if (!isRealized())
{
realize();
}

But the isRealized() function doesn't really say if realize() has been
called or not, it just checks if *any* of the gc's are realized. Our
code happened to call realize on the gc, so the viewer's realize()
never got called. Depending on whether it is allowed to realize gc's
manually it is either a bug or not Smile

Good detective work. We'll need to tighten up the isRealize() function.

Robert.


------------------
Post generated by Mail2Forum
Back to top
Display posts from previous:   
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Similar Topics
Topic Author Forum Replies Posted
No new posts Happy New Year to all OSG Users Chris Hanson General 0 Mon Jan 07, 2019 10:17 am View latest post
No new posts osg-users Digest, Vol 138, Issue 27 Zachary1234 General 0 Tue Jan 01, 2019 1:16 am View latest post
No new posts osg-users Digest, Vol 137, Issue 14 poweruserm@live.com.au General 0 Fri Nov 16, 2018 11:16 pm View latest post
No new posts General Users starting questions in O... A Z General 1 Mon Oct 15, 2018 3:59 am View latest post
No new posts osg-users Digest, Vol 136, Issue 8 jonasydy General 0 Wed Oct 10, 2018 12:40 am View latest post


Board Security Anti Bot Question MOD - phpBB MOD against Spam Bots
Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP