A Machine-learning, artificial-intelligence system designed to flag safety hazards on construction sites has once again provided a backstop for the judges of ENR’s annual photo contest. Known as “VINNIE”— Very Intelligent Neural Network for Insight and Evaluation—the software uses object recognition technologies similar to those in an autonomous vehicle to analyze construction photos and videos for risks. Developed two years ago by Cambridge, Mass.-based Smartvid.io Inc., the system matches pixels in images to a library of objects to automatically tag or note the absence of specific items, such as hardhats, gloves or safety colors. 

In 2016, ENR and several industry firms contributed thousands of construction images to help train and test VINNIE. The system processed unidentified images from ENR’s photo-contest database to help launch the software. The product’s basic level of hazard-spotting intelligence also was introduced as a free public utility in 2017.

VINNIE’s accuracy and capabilities have improved since it was deployed in ENR’s 2016 Year in Construction Photo Contest. In addition to scanning for hardhats and safety colors, VINNIE now sees gloves and eyewear. After looking at 763 images submitted to this year’s contest, VINNIE found 97 people not wearing gloves, 31 without hardhats, 89 without safety colors and 11 without eyewear. In each case it is up to human experts to decide whether those instances depict actionable risk. 

VINNIE flagged two photos chosen by ENR’s human judges as finalists. One features a gloveless worker whose hands are partially obscured. The other shows two ironworkers without high-visibility vests. ENR decided to publish both photos after conferring with this year’s contest safety judge, Keith Snead, safety director at Limback Holdings Inc. Snead says that while the workers “may be violating site-specific requirements set by general contractors,” they aren’t violating Occupational Safety Health Administration regulations. 

Josh Kanner, Smartvid.io’s founder and CEO, says detecting gloves is “an important addition,” because several of his customers have adopted 100% glove compliance policies “due to the high prevalence and cost of hand lacerations.” Hand injuries are one of the leading nonfatal injuries involving days away from work in private industry, according to the U.S. Bureau of Labor Statistics. 

Boston-based contactor Suffolk—which uses Smartvid.io on many of its projects—began requiring gloves about 18 months ago. Marty Leik, Suffolk’s regional safety manager, says VINNIE has helped improve personal protective equipment (PPE) compliance rates among subcontractors. Leik, who helped judge ENR’s 2016 photo contest, uses VINNIE to identify specific jobsite areas and any trades that are not 100% PPE compliant. “That allows us to engage that trade partner and follow up with them to implement improvements,” says Leik, who stressed that VINNIE isn’t used to punish individuals but is used “strictly for improvement and the benefit of everyone on the site.” 

Leik would like VINNIE to recognize harnesses and tethers as well as fall exposures such as open holes or missing guardrails, items Kanner says are more challenging to detect. “You need to know if there’s an edge before you alert that there’s a missing guardrail, versus this is just two columns that have nothing in between them,” Kanner says. “Things like a span between two columns missing a guardrail requires a lot of context.” 

Kanner says VINNIE will be able to recognize slip, trip and fall hazards such as debris and disorderly piles of materials by the end of Q1.

But because the definition of a “messy jobsite” is “ambiguous,” Kanner says VINNIE won’t just flag an area for merely having a certain number of errant coffee mugs on the ground; rather VINNIE is learning to “look at everything in an image and decide for itself” if the area poses a threat, Kanner says.

Smartvid.io has also integrated with several other systems during 2017, such as Autodesk’s BIM 360 Field. An integration with OxBlue’s construction time-lapse cameras recently went live and will be formally announced later in Q1. “The integration takes less than 90 seconds for the user to set up,” Kanner says, “and then data starts flowing from one or many OxBlue cameras on your site into the Smartvid.io platform.”

Firms are starting to use VINNIE for more than just safety. Clayco Inc. is streamlining progress tracking by comparing images uploaded to Smartvid.io with actuals on jobsites. Voice tags in the field help to automatically align content in the Smartvid.io app with specific keywords for machine learning for voice analytics. 

Clayco also is cataloging years of imaging data based on content categories such as “tilt-up construction” so it can easily be recalled for marketing and other purposes, “as opposed to having this content idly laying on servers and hope that somebody one day can find it,” says Tomislav Žigo, Clayco’s vice president for virtual design and construction.

Žigo ultimately imagines using VINNIE to recognize building components such as concrete and rebar and to optimize site logistics by analyzing potential site hazards and egress routes. Perhaps VINNIE could even flag a worker for using a tool in a manner that could cause an injury.

“Anything that can be automatically recognized and tagged,” Žigo says, “we find useful.”

Related: ENR's "2017 Year in Construction" Photo Contest Winners