During the construction of a housing project in Martha’s Vineyard, Massachusetts last winter, a 32 year-old worker named Jose Luis Collaguazo Crespo was injured. He fell to his death from the basement after slipping off a ladder on the second-floor. He was one of the more than 1,000 construction workers who die each year on the job in the US. This makes it the most dangerous industry for fatal slips and trips.
[how] “Everyone speaks about[how]safety is the number one priority,” said entrepreneur and executive Philip Lorenzo during a presentation on Construction Innovation Day 2025 at the University of California Berkeley in April. “But maybe internally it’s not a high priority. People make shortcuts at work. There’s a tug-of war between… safety, and productivity.
This means that any safety risk that the software flags is accurate and related to a specific OSHA rule. Lorenzo says that the software was launched in October 2024 and is now being used on hundreds of construction site in the US. Versions specific to building regulations in Canada, the UK and South Korea have also been implemented. Safety AI
is one of many AI construction safety tools which have emerged in the last few years, from Silicon Valley to Hong Kong to Jerusalem. Many of these tools rely on human “clickers” in low-wage nations to manually draw bounding box around images of key items like ladders in order to label huge volumes of data for training an algorithm.
Lorenzo claims Safety AI is the only one that uses generative AI to flag violations. This means an algorithm which can do more than just recognize ladders and hard hats. The software can “reason”or analyze, what is happening in an image and determine if there is an OSHA violation. Lorenzo says that this is a more sophisticated form of analysis than object detection, which is the industry standard. Safety AI, as evidenced by its 95% success rate, is not an all-knowing and flawless intelligence. It requires a safety inspector with experience to oversee the system.
Visual language model in real life
Robots, AI, and other technologies thrive in controlled environments that are largely static, such as factory floors or shipping terminals. Construction sites, by definition, change a little every day.
Lorenzo believes he has built a better method to monitor sites using a type generative AI known as a visual language models, or VLM. A VLM is a LLM with a visual encoder that allows it to “see”images of the world, and analyze what’s going on in the scene.
Lorenzo’s group has compiled what he calls “a golden data set” of OSHA violations using years of reality capture images gathered from customers with their explicit consent. He has carefully gathered this data over the years and is not concerned that a billion dollar tech giant could “copy and crush him”.
Lorenzo had a smaller team consisting of construction safety experts ask strategic questions to the AI. The trainers feed test scenes from the golden set of data to the VLM, and then ask questions that guide it through the process. If the VLM does not generate the correct response, for example, it registers a false negative or misses an offense, the human trainers will go back and adjust the prompts or inputs. Lorenzo says the VLM learns “how to think a certain way” and can therefore draw subtle conclusions from an image.
COURTESY DRONEDEPLOEY
Lorenzo says that VLMs are better than older methods in analyzing ladder use, which is responsible for 24% of fall deaths.
Lorenzo says that traditional machine learning makes it difficult to answer a question like “Is someone using a ladder in an unsafe manner?” “You can locate the ladders. You can find them. Only the VLM is able to logically reason, and say, “Well, that’s fine” or “Oh no, they’re on the top rung.” Here’s an OSHA reference that says it’s not safe to be on the top step. Are they using it as stilts ? Combining these factors will determine if the ladder in the image is being used safely. Lorenzo says that the system requires over a dozen layers to arrive at a conclusion. DroneDeploy does not release its data to be reviewed, but Lorenzo says he wants his methodology to be independently auditedby safety experts.
Missing 5%
While using vision language models to improve construction AI is promising, there are still “some pretty fundamental issues”, including hallucinations, and anomalous hazards that the VLM was not trained for, says Chen Feng. He is the director of New York University’s AI4CE Labwhich develops technologies in 3D mapping, scene understanding, construction robotics, and other areas. Safety AI’s success rates are 95%, which is a good number. But how can we improve the remaining 5%?
Feng cites a paper from 2024 called “Eyes Wide Shut ?”– written by Shengbang Tong, a PhD student in NYU and coauthored with AI luminary Yann LeCun — that noted “systematic flaws” in VLMs. Feng says that VLMs can perform at a human-level level for object detection. “But for more complex things, these capabilities still need to be improved.” Feng notes that VLMs struggle to interpret 3D scenes from 2D images. They also lack situational awareness when reasoning about spatial relationships and often lack a “commonsense” about visual scene.
Lorenzo admits that LLMs have “some major flaws,” and that they have difficulty with spatial reasoning. Safety AI uses some older machine-learning techniques to create spatial models of construction site. These methods include segmenting images into key components and photogrammetry – a technique that creates a 3D digital image from a 2D picture. Safety AI has been trained extensively in 10 different problem areas, including ladder use, to anticipate the most frequent violations.
Lorenzo admits that there are still edge cases which the LLM may not be able to detect. He notes that having an additional set of digital eyes is still a benefit for safety managers who are responsible for up to 15 sites at a time.
Aaron Tan is a concrete project supervisor based in San Francisco Bay Area. He says that a tool such as Safety AI can be useful for overworked safety managers. They will save time if they receive an email alert instead of having to drive two hours to visit a particular site. He believes that workers will eventually accept the software if it can be shown to help keep people safe. Tan points out that workers are also concerned that these tools could be used as “bossware”to get them into trouble. “At my previous company, we implemented cameras as a security system [as] . “And the guys didn’t appreciate that,” he says. They were like, “Oh, Big Brother.” You guys are always monitoring me–I don’t have any privacy.’ He says that the “old computer vision,” which is based on machine-learning, “is still better” because it’s a hybrid between machine and human intervention in dealing with deviation. To train the algorithm for a new category, his team gathers a large amount of labeled footage relating to the specific danger and then optimizes it by trimming false negatives and positives. Paz says that the process can take from a few weeks to more than six months.
After the training is complete, Safeguard AI performs an assessment of risk to identify potential hazards at the site. It can “see”in real-time, the site by utilizing footage from any nearby camera that is connected to the internet. It then uses an AI agent that sends instructions to site managers via their mobile devices. Paz refuses to provide a price, but says that his product is only affordable for builders in the “mid-market”, specifically those who manage multiple sites. The tool is used at approximately 3,500 sites across Israel, the United States and Brazil.
Buildots is a Tel Aviv-based company that MIT Technology Review profiled in 2020. Instead of doing safety analyses, they create visual progress reports for sites every week or two weeks. Buildots uses an older method of machine-learning with labeled data. Roy Danon, CEO of Buildots, says that the system must be 99% accurate. “We cannot have any hallucinations,” he adds.
According to him, gaining labeled data for training is easier now than when he and cofounders started the project in 2018. This is because video footage from sites allows each object, like a socket, to be captured and labeled at many different frames. The tool is expensive–about fifty builders, many with revenues over $250 million, use Buildots across Europe, the Middle East and Africa, Canada and the US. It has been used in over 300 projects.
Ryan Calo is a specialist in robotics, AI law, and construction safety at the University of Washington. He likes the idea of AI to improve construction safety. Calo is concerned that, because experienced safety managers in construction are already scarce, builders may be tempted to automate the safety process. “I think AI and Drones are super smart for spotting safety issues that would otherwise kill employees,” he says. “As long as it is verified by a human.”
Andrew Rosenblum, a freelance tech reporter based in Oakland CA.
