Access to Rendered Frame buffer of undocked window

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Tue Jun 02, 2020 4:01 am

Greetings,

Following are my objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels

I took a look at two samples provided in the supplied PDK
A. CameraPDK - Where I am facing the following limitations
Even if I get handle to that window handle possibly using Window services of PDK P3D::IWindowV440 and the details of the custom camera being rendered in it using GetCameraSystem(), there are no interfaces which I can implement to get the access to rendered frame buffer.


B. OpenCVStereoCamera - There are two limitations I am facing there
1. The OpenCVInterface is subscribed to a IVRPlugin interface which limits me from opening a custom camera mentioned in the aircraft.cfg file
2. The openCVRenderEffect plugin which implements OpenCVRenderEffect::UpdateImageData() prevents capturing the DX11 framebuffer from an undocked window

SimConnect APIs are a much more higher level abstraction and does not allow me to tap into the rendering service.

I tried searching the forum an the learning center but could not gather much.

I am in need of urgent help where I am able to meet the objectives mentioned at the beginning of this post. Awaiting for an earliest reply at the earliest.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Wed Jun 03, 2020 9:23 am

Any suggestions please.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

ztzhan
Posts: 1
Joined: Wed Jun 03, 2020 6:45 pm

Re: Access to Rendered Frame buffer of undocked window

Postby ztzhan » Wed Jun 03, 2020 6:48 pm

Hi

I think this is a great question. I am also playing around with the OpenCV example but now I have some confusion. I've successfullly loaded the plugin on p3d but I don't know how to use it as I didn't find any usage information of this example. After I launched p3d the plugin was loaded but nothing happens when I enabled VR. Do you know how its functionality can be intrigued in p3d?

Thanks in advance

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Thu Jun 04, 2020 6:22 am

@ztzhan

You have to follow the code with the OpenCV_DLLMain.cpp file from the DLL Start function. Any callback service that is subscribed to will call the ::Invoke interface method (during an event) implemented in the class which inherits the PDK service classes.

In my opinion you will be able to get the VR stream out once the VR headset is connected and you enable the VR HMD.

Unfortunately this is what I have concluded from the OpenCVStreoCamera example.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Thu Jun 04, 2020 11:30 am

I further investigated but could not succeed in getting the Undocked Window displaying a camera from aircraft.cfg rendering into OpenCV Mat.
Can someone please help.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Fri Jun 05, 2020 4:46 am

Greetings,

This issue is posing as a major roadblock in our development. Requesting urgent help from the LM P3D team.

Thanks in advance.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

User avatar
Rob McCarthy
Lockheed Martin
Posts: 2878
Joined: Wed Aug 24, 2011 1:37 pm

Re: Access to Rendered Frame buffer of undocked window

Postby Rob McCarthy » Fri Jun 05, 2020 12:26 pm

Hi Shouvik,

Have you looked at the available Rendering Services in the PDK? The D3D12Texture sample shows how to use these services to draw into an existing view.

Regards,
Rob McCarthy
Rob McCarthy
Prepar3D® Core Lead

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Fri Jun 05, 2020 4:25 pm

Thanks Rob for the prompt reply.

I am actually not trying to overlay/draw any texture on an existing view. I am looking forward to accomplishing the following three objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels

In summary I have a custom camera with a mount location, FoV,Sensor mode etc. defined in the aircraft.cfg file and I need to get that very view (undocked) into OpenCV Mat object.

Hope I could explain it properly
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

shouvik1984
Posts: 14
Joined: Mon Apr 20, 2020 8:46 am

Re: Access to Rendered Frame buffer of undocked window

Postby shouvik1984 » Wed Jun 10, 2020 12:07 pm

Any insights yet. Need urgent help.
Best Regards
Shouvik Das

P3Dv4.4 Windows 10 x64 Core i7-8700K GTX1080Ti 32GB DDR 3600MHz

User avatar
Beau Hollis
Lockheed Martin
Posts: 2167
Joined: Wed Oct 06, 2010 3:25 pm

Re: Access to Rendered Frame buffer of undocked window

Postby Beau Hollis » Thu Aug 20, 2020 1:10 pm

You would need to create a post process effect in order to get access to the windows render target. From there you would need to create your own readback buffer and copy the texture into it. Our OpenCV sample does what you need in revers, i think, so it should at least provide the info you need regarding the proper image format and memory layout for reading to/from a DX RGBA8 texture.

Alternatively, we have SimConnect functions for saving the contents of any view to an image or streaming it over the network. In that case, we use a similar internal process where we readback the rt image a frame later and run it through encoding in background job. We don't currently provide PDK access to our internal request system for reading back frame data.
Beau Hollis
Prepar3D Software Architect


Return to “Software Development Kit (SDK) Questions”

Who is online

Users browsing this forum: No registered users and 19 guests