Can you please tell me how many video playback/live streams you are checking at the IVS client when this issue occurs ?
Also can you please provide the IPC , the VCN and the IVS firmware version?
What codec you use and how the video is the configured on the IPC side and what is on the IVS client configuration?
(go to Settings -> Video/audio/image -> Video settings on the IPC web interface to check).
if you check the web interface live stream of the camera where you have the IVS client installed, do you have this issue with frame rate?