Access to Rendered Frame buffer of undocked window
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Access to Rendered Frame buffer of undocked window
Greetings,
Following are my objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels
I took a look at two samples provided in the supplied PDK
A. CameraPDK - Where I am facing the following limitations
Even if I get handle to that window handle possibly using Window services of PDK P3D::IWindowV440 and the details of the custom camera being rendered in it using GetCameraSystem(), there are no interfaces which I can implement to get the access to rendered frame buffer.
B. OpenCVStereoCamera - There are two limitations I am facing there
1. The OpenCVInterface is subscribed to a IVRPlugin interface which limits me from opening a custom camera mentioned in the aircraft.cfg file
2. The openCVRenderEffect plugin which implements OpenCVRenderEffect::UpdateImageData() prevents capturing the DX11 framebuffer from an undocked window
SimConnect APIs are a much more higher level abstraction and does not allow me to tap into the rendering service.
I tried searching the forum an the learning center but could not gather much.
I am in need of urgent help where I am able to meet the objectives mentioned at the beginning of this post. Awaiting for an earliest reply at the earliest.
Following are my objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels
I took a look at two samples provided in the supplied PDK
A. CameraPDK - Where I am facing the following limitations
Even if I get handle to that window handle possibly using Window services of PDK P3D::IWindowV440 and the details of the custom camera being rendered in it using GetCameraSystem(), there are no interfaces which I can implement to get the access to rendered frame buffer.
B. OpenCVStereoCamera - There are two limitations I am facing there
1. The OpenCVInterface is subscribed to a IVRPlugin interface which limits me from opening a custom camera mentioned in the aircraft.cfg file
2. The openCVRenderEffect plugin which implements OpenCVRenderEffect::UpdateImageData() prevents capturing the DX11 framebuffer from an undocked window
SimConnect APIs are a much more higher level abstraction and does not allow me to tap into the rendering service.
I tried searching the forum an the learning center but could not gather much.
I am in need of urgent help where I am able to meet the objectives mentioned at the beginning of this post. Awaiting for an earliest reply at the earliest.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
Any suggestions please.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Re: Access to Rendered Frame buffer of undocked window
Hi
I think this is a great question. I am also playing around with the OpenCV example but now I have some confusion. I've successfullly loaded the plugin on p3d but I don't know how to use it as I didn't find any usage information of this example. After I launched p3d the plugin was loaded but nothing happens when I enabled VR. Do you know how its functionality can be intrigued in p3d?
Thanks in advance
I think this is a great question. I am also playing around with the OpenCV example but now I have some confusion. I've successfullly loaded the plugin on p3d but I don't know how to use it as I didn't find any usage information of this example. After I launched p3d the plugin was loaded but nothing happens when I enabled VR. Do you know how its functionality can be intrigued in p3d?
Thanks in advance
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
@ztzhan
You have to follow the code with the OpenCV_DLLMain.cpp file from the DLL Start function. Any callback service that is subscribed to will call the ::Invoke interface method (during an event) implemented in the class which inherits the PDK service classes.
In my opinion you will be able to get the VR stream out once the VR headset is connected and you enable the VR HMD.
Unfortunately this is what I have concluded from the OpenCVStreoCamera example.
You have to follow the code with the OpenCV_DLLMain.cpp file from the DLL Start function. Any callback service that is subscribed to will call the ::Invoke interface method (during an event) implemented in the class which inherits the PDK service classes.
In my opinion you will be able to get the VR stream out once the VR headset is connected and you enable the VR HMD.
Unfortunately this is what I have concluded from the OpenCVStreoCamera example.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
I further investigated but could not succeed in getting the Undocked Window displaying a camera from aircraft.cfg rendering into OpenCV Mat.
Can someone please help.
Can someone please help.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
Greetings,
This issue is posing as a major roadblock in our development. Requesting urgent help from the LM P3D team.
Thanks in advance.
This issue is posing as a major roadblock in our development. Requesting urgent help from the LM P3D team.
Thanks in advance.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
- Rob McCarthy
- Lockheed Martin
- Posts: 3703
- Joined: Wed Aug 24, 2011 1:37 pm
Re: Access to Rendered Frame buffer of undocked window
Hi Shouvik,
Have you looked at the available Rendering Services in the PDK? The D3D12Texture sample shows how to use these services to draw into an existing view.
Regards,
Rob McCarthy
Have you looked at the available Rendering Services in the PDK? The D3D12Texture sample shows how to use these services to draw into an existing view.
Regards,
Rob McCarthy
Rob McCarthy
Prepar3D® Core Lead
Prepar3D® Core Lead
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
Thanks Rob for the prompt reply.
I am actually not trying to overlay/draw any texture on an existing view. I am looking forward to accomplishing the following three objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels
In summary I have a custom camera with a mount location, FoV,Sensor mode etc. defined in the aircraft.cfg file and I need to get that very view (undocked) into OpenCV Mat object.
Hope I could explain it properly
I am actually not trying to overlay/draw any texture on an existing view. I am looking forward to accomplishing the following three objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels
In summary I have a custom camera with a mount location, FoV,Sensor mode etc. defined in the aircraft.cfg file and I need to get that very view (undocked) into OpenCV Mat object.
Hope I could explain it properly
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
-
- Posts: 17
- Joined: Mon Apr 20, 2020 8:46 am
Re: Access to Rendered Frame buffer of undocked window
Any insights yet. Need urgent help.
Best Regards
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
Shouvik Das
P3Dv5.3 Windows 10 x64 i7-10850H Quadro RTX3000 32GB DDR 2933MHz
- Beau Hollis
- Lockheed Martin
- Posts: 2452
- Joined: Wed Oct 06, 2010 3:25 pm
Re: Access to Rendered Frame buffer of undocked window
You would need to create a post process effect in order to get access to the windows render target. From there you would need to create your own readback buffer and copy the texture into it. Our OpenCV sample does what you need in revers, i think, so it should at least provide the info you need regarding the proper image format and memory layout for reading to/from a DX RGBA8 texture.
Alternatively, we have SimConnect functions for saving the contents of any view to an image or streaming it over the network. In that case, we use a similar internal process where we readback the rt image a frame later and run it through encoding in background job. We don't currently provide PDK access to our internal request system for reading back frame data.
Alternatively, we have SimConnect functions for saving the contents of any view to an image or streaming it over the network. In that case, we use a similar internal process where we readback the rt image a frame later and run it through encoding in background job. We don't currently provide PDK access to our internal request system for reading back frame data.
Beau Hollis
Prepar3D Software Architect
Prepar3D Software Architect