Hikvision DVR - cameras stop recording

[Upper Atmosphere]

We have a pair of DS-7332HUI-K4 DVRs, one hosts 8 IP and 29 analog cameras, the other is exclusively IP.

We have had a couple of issues when cameras have stopped recording altogether. There will be a cluster of "Lost SYN - rtsp data lost sync" warnings in the log, then most will pick back up and resume normal operations. We had one instance on the 6th where all but 11 cameras stopped recording (everything on the DVR with analog cameras, the second DVR had some that kept going). Some of the cameras still registered motion detection, but a Server Status Information report from the logs about a half hour later showed the time on all of the cameras was the time they stopped recording. The cameras on the second DVR showed nothing out of the ordinary at the time the others stopped, I was expecting a brownout or something though all of the equipment is on battery backup.

The analog cameras are cabled directly into the DVR, the IP cameras go into a POE switch (which is not attached to the normal network) and the DVR is connected to the switch. I can't imagine it is a network configuration issue since these are all on a separate network, but I am going to try locating a different POE switch - I could see it having an effect on the IP cameras but not the analog cameras. I don't see anything in the logs indicating that recording has stopped, other than the timestamps during the Server Status Information updates.

Any thoughts?

 

[SouthernYankee]

1) look at the cables between the DVR and switch.
2) make sure the NVR/ DVR and switch are on a ups.
3) do a power off reset of everything.

Is there log files that indicate if the camera all dropped at once.

 

[alastairstevenson]

the IP cameras go into a POE switch (which is not attached to the normal network)

Assuming that the NVR LAN does not have it's own access to the internet ...
Some questions / thoughts :

What other equipment, if any, is also attached to the switch?
Does the switch have any protective features such as broadcast storm control or jabber isolation?
Does the switch have any logs that might indicate a problem around the times in question?
Was there any work being done on the NVR LAN that could have created something like a network loop?

 

[Upper Atmosphere]

Already did 2 and 3 (they were set up on a UPS in the first place), the cabling between the DVR and switch is a fresh patch cable. Patch cables are cheap, so I'll swap it out and see what happens.

The IP camera switch has nothing but IP cameras attached to it. It uplinks to another POE switch which then goes to the DVRs. I have checked the logs on both switches, and neither one showed anything out of the ordinary. I'll replace the uplink cable between the switches as well (like I said, cable is cheap and these aren't using fiber uplinks). Neither switch has anything funky going on with configuration, no VLANs or anything else since they do not communicate with the main network. We have had the network setup running since summer, several weeks prior to this (mid November) we had an NVR replaced with another DVR to increase the number of channels. It had been running adequately up until 12/6. No other work was being performed. The DVRs and the second POE switch were introduced late summer, replacing an American Dynamics unit.

I also do not see anything out of the ordinary in the DVR logs - I see motion detection alarm starts and stops as usual, then at 22:14:34 there is a Lost SYN - rtsp data lost sync warning on two of the IP cameras. Immediately after that, motion detection alarm starts and stops resume. At 22:27:45 the system registers a Server Status Information event, all of the analog cameras have a time stamp of 22:14:26 and the eight IP cameras have time stamps within ten seconds of that.
The only other events registered (apart from motion detection) are Remote Operation Login/Logout, these events are on the same second each time and come from a system with IVMS-4200 software used to view the camera feed. A review of the client logs showed nobody actually using it at the time.

The first DVR with the analog cameras started recording again immediately after a reboot (this was a software reboot). The second DVR required a power down reboot before it began recording again, though camera feed was visible after a software reboot.

 

[Upper Atmosphere]

Spent a little time in the server room looking at things, rerouting a couple, and - Lord willing, the creek don't rise, and the cables reach - should be able to eliminate the old POE switch entirely. We'll see if that makes any difference. Don't see how it would make a difference with the analog cameras, but I think the simpler the setup the better in this case.

 

[SouthernYankee]

It is very strange that both DVR/NVR have problems at the same time. That looks like a power problem of some sort. Do the IP cameras have any onboard logging capability. Also do the problems seem to occur nightly at the same time.