OpenSceneGraph Forum Forum Index OpenSceneGraph Forum
Official forum which mirrors the existent OSG mailing lists. Messages posted here are forwarded to the mailing list and vice versa.
 
   FAQFAQ    SearchSearch    MemberlistMemberlist    RulesRules    UsergroupsUsergroups    RegisterRegister 
 Mail2Forum SettingsMail2Forum Settings  ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
   AlbumAlbum  OpenSceneGraph IRC ChatOpenSceneGraph IRC Chat   SmartFeedSmartFeed 

Synchronizing with textures uploads.


 
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General
View previous topic :: View next topic  
Author Message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Wed Mar 07, 2018 9:03 pm    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi,

I'm trying to build an augmented reality application using OSG. My main problem, with which I've been struggling for weeks, hacking little temporal victories only to find myself in the ground again, is keeping in sync the camera position with the video frame uploaded to a background texture.

When a new video frame is processed, I call

Code:
osg::ImageStream::setImage(img.cols, img.rows, 1, GL_RGBA8, GL_BGRA, GL_UNSIGNED_BYTE,   img.data, osg::Image::NO_DELETE, 1);


and post in a queue the camera matrix to be read the next rendering cycle by the thread in which OSG runs. The camera position is updated, but the video frame may not make it in time to upload to GPU for this rendering frame, and the old image corresponding to the previous video frame appears instead.

From what I've noticed, the call to ImageStream::setImage() merely marks the texture as dirty, but doesn't initiate an upload to GPU. So I have the new video frame ready, the GPU is free, and I don't know how to convince it to start uploading. By the time renderingTraversals() is called, the GPU can't load it in time.

This is my first time using OSG and I don't know its tricks. I would appreciate very much any help from the community.

Thank you!

Cheers,
Altin[/code]
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12094

PostPosted: Thu Mar 08, 2018 8:44 am    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi Altin,

The ImageStream::setImage() will increment the modified count on the
Image and force an download of the data to the GPU on the next time
the Texture it's attached to is updated, so as long as you do the
update of the image before the draw traversal you should see it update
in time.

If the image doesn't appear as you'd expect then it sounds like the
draw traversal is happening just before the update. I have no
knowledge of the code you have used to read the video image, do the
setImage, any threading used, and how this coordinates with the frame
loop so there really isn't much I can advise about how to fix things.

Would it be possible to post the code? Or a small example that
illustrates what you are doing.

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 11:20 am    Post subject:
Reply with quote

Hi Robert. Thank you very much for your quick reply.

I use Intel's Threading Building Blocks to create a flow::graph that processes video frames using the OpenCV library. On the final stage I have a frame object that contains both, the image to use as the background as well as the corresponding camera matrix.

Initially, I called ImageStream::setImage() from whatever working thread they were processed (I imagined that if I call this method as soon as possible I'll give GPU time to upload it in the background, and it seemed safe because the ffmpeg plugin also calls it from its thread), and posted a message in a queue to be read by the thread OSG runs and update the camera matrix right before calling renderingTraversals(). But depending on the timing between the worker thread that called ImageStream::setImage and the OSG main thread that updated the matrix, they could or could not update in the same frame.

Next, to not leave tings to chance, I decided to do both, call setImage & update camera matrix from within the OSG thread, just before renderingTraversals:

Code:
    advance(simulationTime);

    eventTraversal();
    updateTraversal();
    update_camera_and_image();
    renderingTraversals();


The transforming matrix takes effect that frame, but the updated texture takes effect on the next frame. (I must say that the images are full hd, and the computer doesn't have the best graphics hardware ever made. I even convert images from BGR to BGRA from my processing pipeline because that way they seem to load a bit faster) So 3d objects move one frame before their siblings from the real world.

I have split these operations and have tried placing them in all sorts of combinations between the calls above, but nothing seems to work. It seems that I need to render a frame just to have the texture uploaded for the next.

And this leads to what I currently do, a very ugly hack, which I'll describe just to show the kind of frustration this has caused me.

Code:

// begining of the rendering loop
imageStream->setImage(frame);
getSceneData()->setNodeMask(0); // avoid rendering 3D scene
// signal GraphicsContext::SwapCallback to not swap the buffers
renderingTraversals(); // render just the (old) background because this triggers the upload of the new video frame to GPU
getSceneData()->setNodeMask(saved_value);// restore 3D scene
update_camera_matrix();
... rest of default OSG loop calls ...
eventTraversal();
updateTraversal();
// signal GraphicsContext::SwapCallback to allow sapping buffers
renderingTraversals();


So I call renderingTraversals() twice in a loop, the first time just to render the background which also triggers the upload of the new image for the next (normal) call to renderingTraversals() where I render the complete scene. And I have to disable / enable swapping buffers by using graphicsContext->setSwapCallback().

For applying came matrix changes I've tried setting a NodeCallback to its transforming node too, but that seems to disable ON_DEMAND rendering which I prefer for the moment.

You mentioned updating the texture, what method should I call and when? The examples I've seen that play video only call ImageStream::setImage()

Thank you!

Cheers,
Altin
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12094

PostPosted: Thu Mar 08, 2018 11:34 am    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi Altin,

Quote:
From your description I can't see any reason why the camera view
matrix and image update have the view direction before the texture
update. However, you've only provide a small glimpse in to what your
who integration code is.

I would recommend not calling renderingTraversals() twice in one
frame, this will just complicate things. Right now you need to work
out what is going on and why, then once you understand things you can
come up with a solution.

First thing I would recommend is reverting back to the simpler frame
loop, the setting the ThreadingModel of of the viewer to
SingleThreaded to avoid any possibility that update and cull
traversals are happening concurrently to the draw dispatch for the
previous frame. The later would likely see the video image updating
before the view matrix so you'd see the image and view direction out
of step, but with the view direction lagging. Could this be what is
actually happening?

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 12:18 pm    Post subject:
Reply with quote

Hi Robert.

This has become such a big (psychological even) problem to me, that I'm ready to rewrite this whole thing from scratch if it need be.

I don't know what code to provide more. This is basically all I do. All I have left out are details on converting matrices from OpencCV to OpenGL, and using a circular_buffer to post true / false booleans that are read by a class that derives from GraphicsContext::SwapCallback to decide whether to allow or not swapping buffers.

I use (the default) DrawThreadPerContext for the moment.

Steeping through the code, It seems that uploading the image is done by applyTexImage_subload() called from within TextureRectangle::apply(state) called from a GraphicsThread. And this operation is triggered by renderingTraversals(). It just doesn't upload in time and the background rectangle drawn by this renderingTraversal() uses the previous image.

Is there any way I can trigger applyTexImage_subload() sooner?
Would it help if I used another camera that also uses the same ImageStream as texture and renders to a small texture, and set its rendering order before that of the camera that renders the background?

I don't know what else to do, other than presuming that this source file is cursed and start creating a new file and rewriting this part of the program from the beginning.

Thank you!

Cheers,
Altin
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12094

PostPosted: Thu Mar 08, 2018 12:52 pm    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi Altin,



On 8 March 2018 at 12:18, Altin Gjata <> wrote:
Quote:
I use (the default) DrawThreadPerContext for the moment.

The first thing you need to try is SingleTheaded:

viewer.setThreadingModel(osgViewer::Viewer::SingleTheaded);

I can't stress this enough, this is the biggest variable to test. I
didn't suggest you do this in my previous email for no reason.

Quote:
Steeping through the code, It seems that uploading the image is done by applyTexImage_subload() called from within TextureRectangle::apply(state) called from a GraphicsThread. And this operation is triggered by renderingTraversals(). It just doesn't upload in time and the background rectangle drawn by this renderingTraversal() uses the previous image.

Is there any way I can trigger applyTexImage_subload() sooner?

The subload will be happening as sooner as is necessary to properly
reflect the updates to the osg::ImageStream, it's a pretty widely
fleshed out part of the OSG, this won't be the problem, the problem
will be elsewhere. If the subload is not happening soon enough for
you then it's because the updated dirty of the image stream isn't
happening soon enough.

Quote:
Would it help if I used another camera that also uses the same ImageStream as texture and renders to a small texture, and set its rendering order before that of the camera that renders the background?

No. Coming up with ever more complicated solutions to a problem you
don't yet understand is only going to take you further away from the
solution.

The key to a solution is understanding the problem. Without actual
code to look or even better test there really isn't too much we can do
apart from give you general pointers.

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 1:07 pm    Post subject:
Reply with quote

Ok, I'll start with the threading model and see where it leads. I haven't tried it earlier because I suspect that with the SingleThreaded mode it will work as expected, but I want to take advantage of multi threading.

Thanks for the patience Robert, I'll try your suggestion and see if I can overcome this, if not I'll just have to live with that ugly hack.

Cheers,
Altin
Back to top
View user's profile Send private message
Voerman, L.
Guest





PostPosted: Thu Mar 08, 2018 1:12 pm    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi Altin,just a few remarks:


you wrote:
 >Steeping through the code, It seems that uploading the image is done by applyTexImage_subload() called from within
 >TextureRectangle::apply(state) called from a GraphicsThread. And this operation is triggered by renderingTraversals(). It just doesn't upload in
 >time and the background rectangle drawn by this renderingTraversal() uses the previous image. That's not how it works, openGL will upload the image before using it to render. This might be slow for a big image, but the old image will not be used. You might be tricked by a tripple buffering driver, where at the next buffer swap you will see a render containing the old image.


2> test singlethreaded. I've seen no hint that you marked the stateset containing the imageStream texture as DYNAMIC,  so the renderingTraversal might return before the texture or uniform has been apply'ed - trusting the (default) STATIC promise. 


Regards, Laurens.


On Thu, Mar 8, 2018 at 1:48 PM, Robert Osfield < (
Only registered users can see emails on this board!
Get registred or enter the forums!
)> wrote:
Quote:
Hi Altin,



On 8 March 2018 at 12:18, Altin Gjata < (
Only registered users can see emails on this board!
Get registred or enter the forums!
)> wrote:
Quote:
I use (the default) DrawThreadPerContext for the moment.

The first thing you need to try is SingleTheaded:

  viewer.setThreadingModel(osgViewer::Viewer::SingleTheaded);

I can't stress this enough, this is the biggest variable to test.  I
didn't suggest you do this in my previous email for no reason.

Quote:
Steeping through the code, It seems that uploading the image is done by applyTexImage_subload() called from within TextureRectangle::apply(state) called from a GraphicsThread. And this operation is triggered by renderingTraversals(). It just doesn't upload in time and the background rectangle drawn by this renderingTraversal() uses the previous image.

Is there any way I can trigger applyTexImage_subload() sooner?

The subload will be happening as sooner as is necessary to properly
reflect the updates to the osg::ImageStream, it's a pretty widely
fleshed out part of the OSG, this won't be the problem, the problem
will be elsewhere.  If the subload is not happening soon enough for
you then it's because the updated dirty of the image stream isn't
happening soon enough.

Quote:
Would it help if I used another camera that also uses the same ImageStream as texture and renders to a small texture, and set its rendering order before that of the camera that renders the background?

No. Coming up with ever more complicated solutions to a problem you
don't yet understand is only going to take you further away from the
solution.

The key to a solution is understanding the problem.  Without actual
code to look or even better test there really isn't too much we can do
apart from give you general pointers.

Robert.
_______________________________________________
osg-users mailing list
(
Only registered users can see emails on this board!
Get registred or enter the forums!
)
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org




------------------
Post generated by Mail2Forum
Back to top
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 1:37 pm    Post subject:
Reply with quote

Thanks for the hints Laurens.

Indeed, I don't use DYNAMIC. I did in the beginning, but then I read that it hurts performance and commented it. This is the fragment of code:
Code:

         bg_image_quad_ = createTexturedQuadGeometry(osg::Vec3(), width, height, image, true, true, false);
         if (auto state = bg_image_quad_->getStateSet()) {
            // as explained here: http://forum.openscenegraph.org/viewtopic.php?t=9004
            // DYNAMIC has effect only when applied to StateSet or Drawable
            // apparently DYNAMIC hurts performance, as said here:
            // https://wiki.openmw.org/index.php?title=Rendering_Architecture
            //state->setDataVariance(osg::Object::DYNAMIC);
            //if (osg::ref_ptr<osg::Texture> texture = dynamic_cast<osg::Texture*>(state->getTextureAttribute(0, osg::StateAttribute::TEXTURE))) {
            //   if (auto myImg = dynamic_cast<ImageStream*>(texture->getImage(0))) {
            //      //texture->setDataVariance(osg::Object::DYNAMIC);
            //   }
            //}
         }
         bg_image_camera_ = create_display_image_camera(*this, bg_image_quad_, true);


I'll try in these directions.

Thank you again.

Cheers,
Altin
Back to top
View user's profile Send private message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 3:18 pm    Post subject:
Reply with quote

Just an update. Calling setDataVariance(osg::Object::DYNAMIC), either (or both) in quad's StateSet or in the texture itself doesn't seem to have any affect.

In SingleThreaded mode it works OK.

If the driver uses triple buffering, can I do anything to disable it?

I'm using OSG 3.5.6. I understand that it's neither the stable release nor the most updated development release, but it's the version that vcpkg installs.
Maybe this is all irrelevant.

The content of the entire source file is below, including the commented out hacks.
Code:

#include "StdAfx.h"
#include <osgViewer/Renderer>
#include "Viewer.h"
#include "../sciter/Window.h"
#include "../opencv/feature_detector.h"


osg::Matrix to_osg_Matrix(cv::Matx44d const& rhs) {
   osg::Matrix      lhs{}; // identity
   for (int row = 0; 4 > row; ++row)
      for (int col = 0; 4 > col; ++ col)
         lhs(row, col) = rhs(row, col);
   return lhs;
}



Viewer::Viewer(osg::ArgumentParser& arguments)
   :   BaseClass(arguments)
   ,   circularFrameBuffer_(2) // 2 element capacity
{
   logging::fmt_print("{} >> Viewer\n", cclock::now());
   configure();
}

Viewer::Viewer(int argc, char* argv[])
   :   Viewer(osg::ArgumentParser(&argc, argv))
{
}

Viewer::~Viewer() {
   release_pending_frames();
   logging::fmt_print("{} << Viewer\n", cclock::now());
}



void Viewer::on_frame(frame_stream_t frame_stream) {
   concurrentFrameQueue_.push(frame_stream);
}



int Viewer::run () {
    setReleaseContextAtEndOfFrameHint(false); // copied from osgViewer::Viewer

   // copied from ViewerBase
    if (!isRealized())
    {
        realize();
    }

    const char* run_frame_count_str = getenv("OSG_RUN_FRAME_COUNT");
    unsigned int runTillFrameNumber = run_frame_count_str==0 ? osg::UNINITIALIZED_FRAME_NUMBER : atoi(run_frame_count_str);

    while(!done() && (run_frame_count_str==0 || getViewerFrameStamp()->getFrameNumber()<runTillFrameNumber))
    {
      //SDL_Delay(100);
      upload_video_frame();
      //SDL_Delay(1000);
      use_and_release_video_frame();
      //SDL_Delay(1000);
        double minFrameTime = _runMaxFrameRate>0.0 ? 1.0/_runMaxFrameRate : 0.0;
        osg::Timer_t startFrameTick = osg::Timer::instance()->tick();
        if (_runFrameScheme==ON_DEMAND)
        {
            if (checkNeedToDoFrame())
            {
                frame();
            }
            else
            {
                // we don't need to render a frame but we don't want to spin the run loop so make sure the minimum
                // loop time is 1/100th of second, if not otherwise set, so enabling the frame microSleep below to
                // avoid consume excessive CPU resources.
                if (minFrameTime==0.0) minFrameTime=0.01;
            }
        }
        else
        {
            frame();
        }

        // work out if we need to force a sleep to hold back the frame rate
        osg::Timer_t endFrameTick = osg::Timer::instance()->tick();
        double frameTime = osg::Timer::instance()->delta_s(startFrameTick, endFrameTick);
        if (frameTime < minFrameTime) OpenThreads::Thread::microSleep(static_cast<unsigned int>(1000000.0*(minFrameTime-frameTime)));
    }


   release_pending_frames();
    return 0;
}


/* Copied from ViewerBase */
void Viewer::frame(double simulationTime) {
    if (_done) return;

    // OSG_NOTICE<<std::endl<<"CompositeViewer::frame()"<<std::endl<<std::endl;

    if (_firstFrame)
    {
        viewerInit();

        if (!isRealized())
        {
            realize();
        }

        _firstFrame = false;
    }
    advance(simulationTime);

    eventTraversal();
    updateTraversal();
   //circular_swap_frames_.push_back(true);
   renderingTraversals();
}


//void Viewer::prerender_uploaded_frame() {
//    Contexts contexts;
//    getContexts(contexts);
//
//    // check to see if windows are still valid
//    checkWindowStatus(contexts);
//    if (_done) return;
//
//    double beginRenderingTraversals = elapsedTime();
//
//    osg::FrameStamp* frameStamp = getViewerFrameStamp();
//    unsigned int frameNumber = frameStamp ? frameStamp->getFrameNumber() : 0;
//
//
//    Scenes scenes;
//    getScenes(scenes);
//
//    for(Scenes::iterator sitr = scenes.begin();
//        sitr != scenes.end();
//        ++sitr)
//    {
//        osgViewer::Scene* scene = *sitr;
//        if (!scene) continue;
//
//        osgDB::DatabasePager* dp = scene->getDatabasePager();
//        if (dp) dp->signalBeginFrame(frameStamp);
//
//        osgDB::ImagePager* ip = scene->getImagePager();
//        if (ip) ip->signalBeginFrame(frameStamp);
//
//        if (scene->getSceneData())
//        {
//            // fire off a build of the bounding volumes while we
//            // are still running single threaded.
//            scene->getSceneData()->getBound();
//        }
//    }
//
//    // OSG_NOTICE<<std::endl<<"Start frame"<<std::endl;
//
//
//    Cameras cameras;
//    getCameras(cameras);
//
//    Contexts::iterator itr;
//
//    bool doneMakeCurrentInThisThread = false;
//
//    if (_endDynamicDrawBlock.valid())
//    {
//        _endDynamicDrawBlock->reset();
//    }
//
//    // dispatch the rendering threads
//    if (_startRenderingBarrier.valid()) _startRenderingBarrier->block();
//
//    // reset any double buffer graphics objects
//    for(Cameras::iterator camItr = cameras.begin();
//        camItr != cameras.end();
//        ++camItr)
//    {
//        osg::Camera* camera = *camItr;
//        osgViewer::Renderer* renderer = dynamic_cast<osgViewer::Renderer*>(camera->getRenderer());
//        if (renderer)
//        {
//            if (!renderer->getGraphicsThreadDoesCull() && !(camera->getCameraThread()))
//            {
//                renderer->cull();
//            }
//        }
//    }
//
//    for(itr = contexts.begin();
//        itr != contexts.end() && !_done;
//        ++itr)
//    {
//        if (!((*itr)->getGraphicsThread()) && (*itr)->valid())
//        {
//            doneMakeCurrentInThisThread = true;
//            makeCurrent(*itr);
//            (*itr)->runOperations();
//        }
//    }
//
//    // OSG_NOTICE<<"Joing _endRenderingDispatchBarrier block "<<_endRenderingDispatchBarrier.get()<<std::endl;
//
//    // wait till the rendering dispatch is done.
//    if (_endRenderingDispatchBarrier.valid()) _endRenderingDispatchBarrier->block();
//
//    for(itr = contexts.begin();
//        itr != contexts.end() && !_done;
//        ++itr)
//    {
//        if (!((*itr)->getGraphicsThread()) && (*itr)->valid())
//        {
//            doneMakeCurrentInThisThread = true;
//            makeCurrent(*itr);
//            //(*itr)->swapBuffers();
//         //glFlush();
//        }
//    }
//
//    for(Scenes::iterator sitr = scenes.begin();
//        sitr != scenes.end();
//        ++sitr)
//    {
//        osgViewer::Scene* scene = *sitr;
//        if (!scene) continue;
//
//        osgDB::DatabasePager* dp = scene->getDatabasePager();
//        if (dp) dp->signalEndFrame();
//
//        osgDB::ImagePager* ip = scene->getImagePager();
//        if (ip) ip->signalEndFrame();
//    }
//
//    // wait till the dynamic draw is complete.
//    if (_endDynamicDrawBlock.valid())
//    {
//        // osg::Timer_t startTick = osg::Timer::instance()->tick();
//        _endDynamicDrawBlock->block();
//        // OSG_NOTICE<<"Time waiting "<<osg::Timer::instance()->delta_m(startTick, osg::Timer::instance()->tick())<<std::endl;;
//    }
//
//    if (_releaseContextAtEndOfFrameHint && doneMakeCurrentInThisThread)
//    {
//        //OSG_NOTICE<<"Doing release context"<<std::endl;
//        releaseContext();
//    }
//
//    _requestRedraw = false;
//}


void Viewer::upload_video_frame() {
   frame_stream_t frameStream;
   if (concurrentFrameQueue_.try_pop(frameStream)) {
      auto frame = std::get<0>(frameStream);
      auto imageStream = std::get<1>(frameStream);
      // set image buffer and dirty flag
      imageStream->setImage(frame);
      //if (auto gc = bg_image_camera_->getGraphicsContext()) {
      //   if (auto go = bg_image_camera_->getRenderer()) {
      //      (*go)(gc);
      //   }
      //}
      // signal to draw the 3D scene
      //osg::ref_ptr<osg::Node> sd = getSceneData();
      //auto mm = sd->getNodeMask();
      //sd->setNodeMask(0);
      //circular_swap_frames_.push_back(false);
      ////renderingTraversals();
      //prerender_uploaded_frame();
      //sd->setNodeMask(mm);
      //logging::fmt_print("frame:{:n}\n", frame->ccc);
      circularFrameBuffer_.push_back(frameStream);
   }
}
void Viewer::use_and_release_video_frame() {
   if (!circularFrameBuffer_.empty()) {
      frame_stream_t   frameStream = circularFrameBuffer_.front();
      circularFrameBuffer_.pop_front();
      auto frame = std::get<0>(frameStream);
      update_tracked_objects(frame);
      //dbg_show_colored_plane(frame->dbg_color_);
      async_finished(frame);
      requestRedraw();
   }
}



void Viewer::update_tracked_objects(frame_t frame) {
   if (auto frameData = dynamic_cast<FeatureDetector::FrameData*>(frame->data.get())) {
      for (auto& matched_pattern: frameData->matched_patterns) {
         if (tracked_) {
            tracked_->setMatrix(to_osg_Matrix(matched_pattern->transform()));
         }
         break;
      }
   }
}


void Viewer::release_pending_frames() {
   frame_stream_t frame_stream;
   while (!circularFrameBuffer_.empty()) {
      frame_stream = circularFrameBuffer_.front();
      circularFrameBuffer_.pop_front();
      async_finished(std::get<0>(frame_stream));
   }
   while (concurrentFrameQueue_.try_pop(frame_stream)) {
      async_finished(std::get<0>(frame_stream));
   }
}


void Viewer::dbg_show_colored_plane(int val) {
   if (0 <= val) {
      float red = (float)val / 255.0;
      if (!dbgPlane) {
         if (auto root = dynamic_cast<osg::Group*>(getSceneData())) {
            const float width = 5, height = 5, distance = -20;
            dbgPlane = osg::createTexturedQuadGeometry(
               osg::Vec3(-width / 2 - 4, height / 2 + 4, distance), osg::Vec3(width, 0, 0), osg::Vec3(0, -height, 0), 0, 0, 1, 1);
            dbgPlane->getOrCreateStateSet()->setMode(GL_LIGHTING, osg::StateAttribute::OFF | osg::StateAttribute::PROTECTED);
         root->addChild(dbgPlane);
         }
      }
      if (dbgPlane) {
         osg::Vec4Array* colours = new osg::Vec4Array(1);
         (*colours)[0].set(red,0.0f,0.0, 1);
         dbgPlane->setColorArray(colours, osg::Array::BIND_OVERALL);
      }
   } else if (dbgPlane) {
      if (auto root = dynamic_cast<osg::Group*>(getSceneData())) {
         root->removeChild(dbgPlane);
         dbgPlane = nullptr;
      }
   }
}



/*
**-----------------------------------------------------------------------------
**
**-----------------------------------------------------------------------------
*/

class RenderGUIEventHandler : public osgGA::GUIEventHandler
{
public:
   RenderGUIEventHandler(Viewer *viewer)
      :   viewer_(viewer)
   {}
   ~RenderGUIEventHandler() {
   }
   // osgGA::GUIEventHandler:
   bool handle(const osgGA::GUIEventAdapter& ea, osgGA::GUIActionAdapter& aa) override {
      switch (ea.getEventType()) {
      case   osgGA::GUIEventAdapter::KEYDOWN:
         switch (ea.getKey()) {
         case   osgGA::GUIEventAdapter::KEY_F8:
            ++viewer_->resizing_mode_;
            viewer_->resizeViewports();
            return true;
         }
         break;
      }
      return   false;
   }

private:
   Viewer   *viewer_;
};




void Viewer::configure() {
   // https://groups.google.com/forum/#!topic/osg-users/s8ESNvqtfag
   osgDB::setLibraryFilePathList(".;D:/programing/vcpkg/installed/x86-windows/tools/osg/osgPlugins-3.5.6");
   osgDB::setDataFilePathList(".;D:/programing/sdk/OSG/OpenSceneGraph-Data;D:/programing/sdk/OSG/data");

   //osg::DisplaySettings::instance()->setNumMultiSamples(4);

   auto main_camera = getCamera();
   main_camera->setClearMask(GL_DEPTH_BUFFER_BIT);

   // 'F7' - cycle between stats
   osg::ref_ptr<osgViewer::StatsHandler>   statsHandler = new osgViewer::StatsHandler();
   statsHandler->setKeyEventTogglesOnScreenStats(osgGA::GUIEventAdapter::KEY_F7);
   addEventHandler(statsHandler);
   // 'F11' - fullscreen / windowed
   osg::ref_ptr<osgViewer::WindowSizeHandler>   sizeHandler = new osgViewer::WindowSizeHandler();
   sizeHandler->setKeyEventToggleFullscreen(osgGA::GUIEventAdapter::KEY_F11);
   addEventHandler(sizeHandler);

   addEventHandler(new RenderGUIEventHandler{this});

   // disable closing window with Esc
   setKeyEventSetsDone(0);


   setRunFrameScheme(osgViewer::Viewer::ON_DEMAND);
   //setRunMaxFrameRate(30);
   setThreadingModel(osgViewer::Viewer::SingleThreaded);
}



osg::ref_ptr<osg::Geometry> Viewer::createTexturedQuadGeometry(const osg::Vec3& pos,float width,float height,
   osg::ref_ptr<osg::Image> image,   bool useTextureRectangle, bool xyPlane, bool option_flip)
{
    bool flip = image->getOrigin()==osg::Image::TOP_LEFT;
    if (option_flip) flip = !flip;

    if (useTextureRectangle)
    {
        osg::ref_ptr<osg::Geometry> pictureQuad = osg::createTexturedQuadGeometry(pos,
                                           osg::Vec3(width,0.0f,0.0f),
                                           xyPlane ? osg::Vec3(0.0f,height,0.0f) : osg::Vec3(0.0f,0.0f,height),
                                           0.0f, flip ? image->t() : 0.0, image->s(), flip ? 0.0 : image->t());

        osg::ref_ptr<osg::TextureRectangle> texture = new osg::TextureRectangle(image);
        texture->setWrap(osg::Texture::WRAP_S, osg::Texture::CLAMP_TO_EDGE);
        texture->setWrap(osg::Texture::WRAP_T, osg::Texture::CLAMP_TO_EDGE);


        pictureQuad->getOrCreateStateSet()->setTextureAttributeAndModes(0,
                                                                        texture,
                                                                        osg::StateAttribute::ON);

        return pictureQuad;
    }
    else
    {
        osg::ref_ptr<osg::Geometry> pictureQuad = osg::createTexturedQuadGeometry(pos,
                                           osg::Vec3(width,0.0f,0.0f),
                                           xyPlane ? osg::Vec3(0.0f,height,0.0f) : osg::Vec3(0.0f,0.0f,height),
                                           0.0f, flip ? 1.0f : 0.0f , 1.0f, flip ? 0.0f : 1.0f);

        osg::ref_ptr<osg::Texture2D> texture = new osg::Texture2D(image);
        texture->setResizeNonPowerOfTwoHint(false);
        texture->setFilter(osg::Texture::MIN_FILTER,osg::Texture::LINEAR);
        texture->setWrap(osg::Texture::WRAP_S, osg::Texture::CLAMP_TO_EDGE);
        texture->setWrap(osg::Texture::WRAP_T, osg::Texture::CLAMP_TO_EDGE);


        pictureQuad->getOrCreateStateSet()->setTextureAttributeAndModes(0,
                    texture,
                    osg::StateAttribute::ON);

        return pictureQuad;
    }
}


osg::ref_ptr<osg::Camera> Viewer::create_display_image_camera(osg::View& view,
   osg::ref_ptr<osg::Drawable> video_quad, bool b_background)
{
   const osg::BoundingBox& bb = video_quad->getBoundingBox();

   osg::ref_ptr<osg::Camera> image_camera{new osg::Camera{}};

   image_camera->setReferenceFrame(osg::Camera::ABSOLUTE_RF);
   image_camera->setAllowEventFocus(false);

   image_camera->getOrCreateStateSet()->setMode(GL_LIGHTING, osg::StateAttribute::OFF);

   image_camera->setGraphicsContext(view.getCamera()->getGraphicsContext());
   image_camera->setViewport(view.getCamera()->getViewport());

   //image_camera->setViewMatrix(osg::Matrixd::identity());
   image_camera->setProjectionMatrix(osg::Matrix::ortho2D(bb.xMin(), bb.xMax(), bb.yMin(), bb.yMax()));

   if (b_background) {
      image_camera->setRenderOrder(osg::Camera::PRE_RENDER);
      image_camera->getOrCreateStateSet()->setMode(GL_BLEND, osg::StateAttribute::OFF);
      //image_camera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
      //image_camera->setClearMask(0);
      image_camera->setClearColor(osg::Vec4(.3f, 0.4f, 0.5f, 1.f));
   } else {
      image_camera->setRenderOrder(osg::Camera::POST_RENDER);
      image_camera->getOrCreateStateSet()->setMode(GL_BLEND, osg::StateAttribute::ON);
      image_camera->setClearMask(GL_DEPTH_BUFFER_BIT);
   }

   image_camera->addChild(video_quad);
   view.addSlave(image_camera, false);

   return image_camera;
}


void Viewer::cameraSetGraphicsContext(osg::Camera* camera, osg::GraphicsContext* gc) {
   if (camera && gc) {
      camera->setGraphicsContext(gc);

      double fovy, aspectRatio, zNear, zFar;
      camera->getProjectionMatrixAsPerspective(fovy, aspectRatio, zNear, zFar);

      osg::GraphicsContext::Traits const *traits = gc->getTraits();
      double newAspectRatio = double(traits->width) / double(traits->height);
      double aspectRatioChange = newAspectRatio / aspectRatio;
      if (aspectRatioChange != 1.0)
      {
         camera->getProjectionMatrix() *= osg::Matrix::scale(1.0/aspectRatioChange,1.0,1.0);
      }

      camera->setViewport(new osg::Viewport(0, 0, traits->width, traits->height));

      GLenum buffer = traits->doubleBuffer ? GL_BACK : GL_FRONT;

      camera->setDrawBuffer(buffer);
      camera->setReadBuffer(buffer);
   }
}


class DrawSciterAndSwap : public osg::GraphicsContext::SwapCallback
{
public:
   DrawSciterAndSwap(Viewer& viewer, std::shared_ptr<Window> window)
      :   viewer_(viewer), window_(window)
   {}
private:
   void swapBuffersImplementation(osg::GraphicsContext* gc) override {
      //bool bSwap = viewer_.circular_swap_frames_.front();
      //viewer_.circular_swap_frames_.pop_front();
      //if (bSwap) {
         glFlush(); // flickering happens without this
         if (auto win = window_.lock()) {
            win->drawWithOpenGL();
         }
         gc->swapBuffersImplementation();
      //}
   }

   Viewer&                     viewer_;
   std::weak_ptr<Window>         window_;
};



class GraphicsContextResizedCallback : public osg::GraphicsContext::ResizedCallback
{
public:
   GraphicsContextResizedCallback(Viewer *viewer)
      :   viewer_(viewer)
   {}
   ~GraphicsContextResizedCallback() {}
private:
   // osg::GraphicsContext::ResizedCallback:
   void resizedImplementation(osg::GraphicsContext *gc, int x, int y, int width, int height) override {
      if (viewer_) {
         viewer_->resizeViewports(width, height);
         auto traits = const_cast<osg::GraphicsContext::Traits*>(gc->getTraits());
         traits->x = x;
         traits->y = y;
         traits->width = width;
         traits->height = height;
      } else {
         gc->resizedImplementation(x, y, width, height);
      }
   }

   Viewer      *viewer_;
};


void Viewer::setGraphicsContext(std::shared_ptr<Window> window) {
   auto graphicsContext = window->getGraphicsContext();
   cameraSetGraphicsContext(getCamera(), graphicsContext);
   graphicsContext->setSwapCallback(new DrawSciterAndSwap(*this, window));

   std::vector<osg::GraphicsContext*>   contexts;
   getContexts(contexts);
   for (auto ctx: contexts) {
      ctx->setClearColor(osg::Vec4f(0.0f, 0.0f, 0.0f, 1.0f));
      ctx->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
      ctx->setResizedCallback(new GraphicsContextResizedCallback(this));
   }

   //main_camera->setProjectionResizePolicy(osg::Camera::ProjectionResizePolicy::FIXED);
   //if (auto gw = dynamic_cast<osgViewer::GraphicsWindow*>(main_camera->getGraphicsContext())) {
   //   gw->setSyncToVBlank(false);
   //}
}


class BgImageResizedCallback : public osg::Image::DimensionsChangedCallback
{
public:
   BgImageResizedCallback(Viewer& viewer)
      :   viewer_(viewer)
   {
   }
   void operator()(osg::Image* image) override {
      viewer_.create_or_update_image_scene(image);
   }
   Viewer& viewer_;
};



void Viewer::create_or_update_image_scene(osg::ref_ptr<osg::Image> image) {
   if (image) {
      if (image != bg_image_) {
         if (bg_image_) {
            bg_image_->removeDimensionsChangedCallback(bg_image_resized_callback_);
         }
         bg_image_ = image;
         if (!bg_image_resized_callback_)
            bg_image_resized_callback_ = new BgImageResizedCallback(*this);
         bg_image_->addDimensionsChangedCallback(bg_image_resized_callback_);
         if (osg::ImageStream *imageStream = dynamic_cast<osg::ImageStream*>(image.get())) {
            imageStream->play();
         }
      }
      float   width = image->s() * image->getPixelAspectRatio();
      float   height = image->t();
      float xMin = 0.f, xMax = 0.f, yMin = 0.f, yMax = 0.f;
      if (auto geom = dynamic_cast<osg::Geometry*>(bg_image_quad_.get())) {
         if (auto coords = dynamic_cast<osg::Vec3Array*>(geom->getVertexArray())) {
            (*coords)[2].x() = width;
            (*coords)[3].x() = width;
            (*coords)[0].y() = height;
            (*coords)[3].y() = height;
            xMin = (*coords)[1].x();
            yMin = (*coords)[1].y();
            xMax = (*coords)[3].x();
            yMax = (*coords)[3].y();
            coords->dirty();
         }
         if (auto tcoords = dynamic_cast<osg::Vec2Array*>(geom->getTexCoordArray(0))) {
            bool flip = image->getOrigin()==osg::Image::TOP_LEFT;
            float   l = 0.0f,
                  b = flip ? image->t() : 0.0,
                  r = image->s(),
                  t = flip ? 0.0 : image->t()
               ;
            (*tcoords)[0].set(l, t);
            (*tcoords)[1].set(l, b);
            (*tcoords)[2].set(r, b);
            (*tcoords)[3].set(r, t);
            tcoords->dirty();
         }
         geom->dirtyGLObjects();
         geom->dirtyBound();
         if (bg_image_camera_) {
            bg_image_camera_->setProjectionMatrix(osg::Matrix::ortho2D(xMin, xMax, yMin, yMax));
         }
         if (fg_image_camera_) {
            bg_image_camera_->setProjectionMatrix(osg::Matrix::ortho2D(xMin, xMax, yMin, yMax));
         }
         resizeViewports();
      } else if (0 < width && 0 < height)
      {
         bg_image_quad_ = createTexturedQuadGeometry(osg::Vec3(), width, height, image, true, true, false);
         if (auto state = bg_image_quad_->getStateSet()) {
            // as explained here: http://forum.openscenegraph.org/viewtopic.php?t=9004
            // DYNAMIC has effect only when applied to StateSet or Drawable
            // apparently DYNAMIC hurt performance, as said here:
            // https://wiki.openmw.org/index.php?title=Rendering_Architecture
            state->setDataVariance(osg::Object::DYNAMIC);
            if (osg::ref_ptr<osg::Texture> texture = dynamic_cast<osg::Texture*>(state->getTextureAttribute(0, osg::StateAttribute::TEXTURE))) {
               if (auto myImg = dynamic_cast<ImageStream*>(texture->getImage(0))) {
                  texture->setDataVariance(osg::Object::DYNAMIC);
               }
            }
         }
         bg_image_camera_ = create_display_image_camera(*this, bg_image_quad_, true);
         //fg_image_camera_ = create_display_image_camera(*this, bg_image_quad_, false);
         resizeViewports();
      }
   }
}


void Viewer::resizeViewports(int availWidth, int availHeight) {
   if (0 > availWidth || 0 > availHeight) {
      std::vector<osg::GraphicsContext*>   contexts;
      getContexts(contexts);
      if (contexts.empty())
         return;
      //assert(0 < contexts.size());
      auto traits = contexts[0]->getTraits();
      availWidth = traits->width;
      availHeight = traits->height;
   }

   if (!bg_image_)
      return;
   //assert(bg_image_);
    float imgWidth = bg_image_->s() * bg_image_->getPixelAspectRatio();
    float imgHeight = bg_image_->t();
   if (0 >= imgWidth || 0 >= imgHeight) {
      imgWidth = 128.f;
      imgHeight = 64.f;
   }

   double imgAspectRatio = imgWidth / imgHeight;
   double availAspectRatio = (double)availWidth / availHeight;

   float vpX, vpY, vpWidth, vpHeight;

   switch (resizing_mode_) {
   case   kOriginalSize:
      vpWidth = imgWidth;
      vpHeight = imgHeight;
      break;
   case   kFitWindow:
      vpWidth = availWidth;
      vpHeight = availHeight;
      break;
   case   kMaintainAspectRatio:
      if (imgAspectRatio < availAspectRatio) {
         vpHeight = availHeight;
         vpWidth = vpHeight * imgAspectRatio;
      } else {
         vpWidth = availWidth;
         vpHeight = vpWidth / imgAspectRatio;
      }
      break;
   }
   vpX = (availWidth - vpWidth)  / 2.0;
   vpY = (availHeight - vpHeight) / 2.0;

   std::vector<osg::Camera*>   cameras;
   cameras.push_back(getCamera());
   if (bg_image_camera_) cameras.push_back(bg_image_camera_);
   if (fg_image_camera_) cameras.push_back(fg_image_camera_);
   for (auto camera : cameras) {
      if (auto viewport = camera->getViewport()) {
         viewport->setViewport(vpX, vpY, vpWidth, vpHeight);
      }
   }
   if (auto camera = getCamera()) {
      double fovy, aspectRatio, zNear, zFar;
      camera->getProjectionMatrixAsPerspective(fovy, aspectRatio, zNear, zFar);
      camera->setProjectionMatrixAsPerspective(fovy, imgAspectRatio, zNear, zFar);
   }
}






...

Thank you!

Cheers,
Altin[/code]
Back to top
View user's profile Send private message
robertosfield
OSG Project Lead


Joined: 18 Mar 2009
Posts: 12094

PostPosted: Thu Mar 08, 2018 3:53 pm    Post subject:
Synchronizing with textures uploads.
Reply with quote

Hi Altin,

On 8 March 2018 at 15:19, Altin Gjata <> wrote:
Quote:
Just an update. Calling setDataVariance(osg::Object::DYNAMIC), either (or both) in quad's StateSet or in the texture itself doesn't seem to have any affect.

In SingleThreaded mode it works OK.

Good to hear that this works. This is important for understanding
what might be going amiss.

My best guess is that previously it was your view matrix that is one
frame behind rather than your texture update, as when you a call set
set the Camera's view matrix affects the current frame's traversal,
but in DrawThreadPerContext the current frame event, update and cull
traversals all can happen in parallel with the previous frames Draw
traversals, so if you update the view matrix it affects the current
frame, but if you also update the Image this is being read by the
previous frames draw traversal. This will result in the video image
and view direction being out of sync.

SingleThreaded or CullDrawThreadPerContext both prevent the next frame
from starting before the draw dispatch is complete so will avoid this
issue. I would also would have expected setting the DataVariance to
DYNAMIC on the StateSet that you attach your texture to would have
also prevented the next frame from starting before the texture was
dispatched, I can only guess that the something wasn't quite right
with the setting in your code.


Quote:
If the driver uses triple buffering, can I do anything to disable it?

I don't have any knowledge of Intel drivers under Windows so can't
help with this.

Quote:

I'm using OSG 3.5.6. I understand that it's neither the stable release nor the most updated development release, but it's the version that vcpkg installs.
Maybe this is all irrelevant.

The content of the entire source file is below, including the commented out hacks.

A quick can through it mostly looks OK, but it's kinda hard reading in
a browser, if possible in future could you post large segments of code
like this an attachment, this will make it easier to read with a the
readers preferred editor.

--

Personally, now you have it working with SingleThreded to stick with
this. Unless you have really large models that have heavy cull and
draw traversals I would expect DrawThreadPerContext to make that much
difference. If you do see performance issues with SIngleThreaded then
I'd recommend working on what the bottlenecks are rather than just
selecting the multi-threaded solution that in your case introduces
issues.

W.r.t performance issues, I would recommend starting a separate thread
to discuss these, thread as in mailing list/forum thread rather than
CPU one :-)

Robert.


------------------
Post generated by Mail2Forum
Back to top
View user's profile Send private message
Tini
Newbie


Joined: 07 Mar 2018
Posts: 7

PostPosted: Thu Mar 08, 2018 4:28 pm    Post subject:
Reply with quote

Ok, thanks Robert. I was under the impression that the single threaded mode was just for testing, but not the one to use for a finished product. However, it can't be any worse than my double rendering per cycle hack, so I'll stick with single threaded.

Update: I tested CullDrawThreadPerContext, DrawThreadPerContext and CullThreadPerCameraDrawThreadPerContext and it works correctly with these modes too.
I don't know how to explain this , but the problem manifests itself only when I don't call setThreadingModel() at all (and I'm overwriting OSG's run() and frame() methods).

Anyway I'm happy with the result. Thank you again Robert and Laurens.

Cheers,
Altin
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    OpenSceneGraph Forum Forum Index -> General All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum

Similar Topics
Topic Author Forum Replies Posted
No new posts Multi-threaded usage of textures anishuz Submission 7 Tue Oct 17, 2017 8:38 am View latest post
No new posts Multiple video textures using ffmpeg ... mmaurus Plugins [osgPlugins] 6 Thu Oct 05, 2017 10:02 am View latest post
No new posts Is it possible to clear just certain ... amudhan79 General 3 Mon Sep 11, 2017 7:48 pm View latest post
No new posts Simulating Light Lobes with Projected... Trajce Nikolov NICK General 3 Thu Aug 11, 2016 9:57 pm View latest post
No new posts Bindless Textures d_a_heitbrink General 2 Fri Jul 15, 2016 2:21 pm View latest post


Board Security Anti Bot Question MOD - phpBB MOD against Spam Bots
Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP