Video surveillance security system is process of automatic recognition and evaluation of critical situations in LC environments in order to object tracking and analysis. In the road and railroad transportation field safety and security are the most examined topics. Recent security activities in the field of railroad transportation propose to implement video surveillance at level crossing LC conditions. When there are enough detected pixels belonging to moving objects then object is tracked. Primarily there are two major kinds of methods to perform visual tracking. The first type is based on target representation and localization and the second type of methods uses filtering and data association. The smart security system starts by detecting, separating, and tracking moving objects shot in the LC. By allowing the identified targets to discard unsafe situations, ideal trajectories are predicted. Then the level of danger of each objective is instantly estimated. The analysis allows also for recognizing dangerous scenarios. Hazard scenarios are checked and evaluated with different real video image sequences.