Streamline was running smoothly but now there is a lag in the streaming timeline (To complete 10 sec in the timeline of capture, it takes 20-25 seconds) even if I'm monitoring one counter. It is capturing data very slowly.
My System Specs :
RAM: 16 GB
Processor : i5 10th gen
Free Space: 100 GB
and No other application is running in parallel.
Hard to say without further information.* Is the problem during or at the start of live capture, or afterwards once the Analysis happens?* Is the problem that the 10s of data is collected, but the U/I is being laggy? Or is it that there is a 10-15s delay before the first data comes, but then the 10s worth of data come smoothly?If the target appears to be the problem:* What is the target you are running on? (OS, hardware type)* How are you connected (over USB or network)?* Is the target device running with heavy load?* Have you tried restarting the target device?* How are you making the capture? Using gatord on the command line, or using the 'Start' tab in the U/I?
Is the problem during or at the start of live capture or afterwards once the Analysis happens?* Is the problem that the 10s of data is collected, but the U/I lags? Or is there a 10-15s delay before the first data comes, but then the 10s worth of data comes smoothly?
--> After 10s of data collection, U/I gets laggy, and lag increases with time, and eventually, it stops collecting data. But I am sure it is not an issue with the Host machine because I have 2 Boards with similar configurations (Khadas Vim3 and Odroid M1) Odroid works fine. Khadas was also working fine. I have been using streamline for kahdas for the last 10 days there was no issue, but all of a sudden board got rebooted, and streamline got lagging. targe board still has 18GB free space, and the Host has 80 GB.If the target appears to be the problem:* What is the target you are running on? (OS, hardware type)
Board: kahdas Vim3
RAM: 4 GB
Workload: Mobilenet Model inference on CPU and Mali-GPU (I have previously captured data on streamline on same board and same workload it was working fine)
* How are you connected (over USB or network)?
LAN Network* Is the target device running with a heavy load?
Medium load
* Have you tried restarting the target device?
Yes* How are you making the capture? Using gatord on the command line, or using the 'Start' tab in the U/I?
Using U/I.
This video shows what is the exact problem. The host system itself is not lagging but the data is.
While recording this video I was not running any workload on the target device (None).
I see. So it gets slower / increasingly laggy over time. Typically, lag like that happens when the connection to the target is saturated, or where the target is configured to produce a significant amount of data (usually, SPE is enabled).To check, you haven't recently made any changes (updated version of Streamline, changed board or different configuration on target) ?It looks like you are connected to a remote target over TCP/IP...* What is the target you are running on? (OS, hardware type, number and type of cpu cores)* Have you tried restarting the target device?* What sort of counter configuration have you requested?* Since you are running gatord on the remote target, what command line options did you pass to launch? Can you launch gatord with the debug flag and capture its output during a short capture, then send it here. To do so:
gatord -d ...your other arguments... 1> gatord.log 2>&1
Actually, i missed your first reply, so missing some of the information that answers some of above, but if you could still answer:* What sort of counter configuration have you requested?* Since you are running gatord on the remote target, what command line options did you pass to launch? * Can you launch gatord with the debug flag and capture its output during a short capture, then send it here.Additionally, if you are happy, can you "Export" one of the capture files that show this lag and send it to us, so I can examine it.Thanks
* What sort of counter configuration have you requested?
For CPU-Big(A73) and CPU-Little(A53) -- L1/L2 cache, CPU stall cycles, For Mali GPU counters for external reads and L2 stalls* Since you are running gatord on the remote target, what command line options did you pass to launch?
I use "sudo ./gatord"
6457.gatord.log
I have attached the gatord log. For some reason, I am not able to upload the capture file so I have uploaded to gdrive : drive.google.com/.../view
Thanks. If I run your capture through our replayer, it renders correctly (no lag), so I don't think there is some bug being triggered in the U/I that is causing the slow down. Looking at the log, as you say, looks like a pretty typical device with small number of cores / counters, so not particularly anything that would cause some processing bottleneck. I wonder, if you run gator in headless mode, if it captures as expected, or if you still see some lag. You can make a local capture (to disk on the target) with something like:
time sudo ./gatord -c configuration.xml -o capture.apc -t 25
I have reinstalled gator and ARM Streamline it still shows the message "Session already in progressFailed to set up Annotations UDS parent listener. Is the socket already in use?" but capture is working fine now.
I think some files might got corrupted/changed due to the reboot. Thank you for the help and support
No worries. Glad it was fixed!That message can be ignored, it is just informative, we should really suppress or change the wording of it...