Technikjournalisten bloggen

See your machine problems with a clearer few – augmented reality

So far augmented reality is mainly known from games like “Pokémon GO”. While augmented reality in “Pokémon GO” is just a funny feature that allows you to catch the little fantasy animals that the game is about on your mobile phone against a real background, Augmented Reality can offer a real added value for the processing industry. Augmented reality is essentially defined by these three characteristics: First it is a combination of virtually generated objects and real environment with partial superimposition. Second there is the possibility of interaction in real time with the computer generated additional objects and third there partly is a three-dimensional relation to virtual and real objects. Author Hanna Reinert spoke to Stephanie Bolid. She is an associate engineer at Rockwell Automation GmbH that produce augmented reality solutions for the production.


Catching a fantasie animal against real background in the game “Pokèmon go”. Foto: Reinert

Why is augmented reality useful for the processing industry?

Stephanie Bolid: One of the most beneficial use cases right now – that we’re seeing by a lot of interests from our customers – is maintenance. For Example if you don’t have any training on how to repair something on your machine. You don’t need any training. You can just use the holo-lens or iPad to work you through the repair. So you can very easily configure it by step-by-step process. For each step it shows you within augmented reality exactly what you need to do, what part of the machine might be failing, what you need to track or any data associating with it. It can show you an animation for example it can turn into a different colour or it can flash or tell you what button you need to press next. Things like these in order to try to identify the problem and then quickly fix it. And that eliminates the need for training because it can work you through the steps and make it really clear to you what you need to do.

Is augmented reality something you have to use with an IIOT?

Stephanie Bolid:  Definitely. I think that’s the whole premise. It’s just another layer on top of the internet of things where all of your devices are connected and you’re getting real time data right away. Augmented reality helps you visualize that data better. So you already have the data. It is in your controller. You know what’s going on with your machine but you don’t have to dig around for it and your excel spreadsheets. It’s shown right on the device and you can see data changing, too, in real time.

Why are augmented reality devices better than displays on the machines to visualize the data?

Stephanie Bolid:  Well, if one person is onside and it has the holo-lens on, it can be watched remotely by anyone else, who can be any distance away. They can see what the person sees through the glasses just on their monitor. So for example there’s somebody who can be walking around and the person that is remote can give commands like “oh I really like to see what the machine is doing in the room on your right side.”

Stephanie Bolin using the holo-lens to log in into the virtual menu. Foto: Reinert

So the holo-lens is also a camera?

Stephanie Bolid:  Yes, you can do a livestream. And so somebody who is remote can check exactly what they’re doing on their monitor. You can also record and if you have something that’s pre-recorded on the holo-lens you can download that to the device. Then multiple people can watch it and it’ll be like a training. It could be an instruction like “okay this is what you going to be doing today. Take this 10 minutes training and it walks you through all you need”. After watching and trying the recorded training you pass it off to the next person. That eliminates the need for very extensive or expensive training.

Do you think that augmented reality offers new opportunities to further develop artificial intelligence or connect ar with ai?

Stephanie Bolid:  I believe so, augmented reality has a lot of capabilities that we haven’t even explored yet. Right now we’re doing just a really basic application to show how to visualize certain devices and what they’re doing like for example being able to detach the infra-red light that is configured in the safety centre. We can visualize that on the holo-lens. All applications we have right now are very basic but the technology offers so much capabilities and I definitely believe that you can somehow interface it with artificial intelligence as well.

Is there a need that you can see the safety center on the holo-lens? You could paint the ground as well.

Stephanie Bolid:  You definitely could paint it on the ground but it’s easily configured. For example if something changes on your machine or you replace it, you don’t need to repaint the ground. Because in the augmented reality it’s just a couple of clicks to replace it there. So it is a lot more efficient and also you don’t need to buy paint and hire somebody to paint the ground.

Stephanie Bolin checking the safety centre through the holo-lens. Foto Reinert

Do you need a pair of augmented reality glasses for each worker?

Stephanie Bolid:  Well, I think it is not something you’ll see like everyone wears it all day for their shift. It is more like one person who will walk out on the floor and then be wearing it. So that person can do a scan of the floor having a different data associate for the machines and they will all pop up like a bubble over the machine. Also it’s definitely a short time thing. You don’t want to be wearing it for eight hours straight. It gets a little heavy on your head and you might lose your balance a little bit. It is for short term to control your machine. Visualizing the data for maybe once every hour for a minute to see what parts of the machine are up in running, what parts don’t run as well or are not as productive. Then you will be able to identify those problems and get in the machine to see what’s wrong and how you can fix it.

Do you need any requirements for using augmented reality?

Stephanie Bolid: Yes, all we do with the augmented reality is reading the information from the controller. So we actually don’t need any new sensors for the augmented reality but we’re reading the data from the controller and configure it using a software which takes the data from the controller and puts it into a media studio which is value to configure all of the actual augmented elements. So you can bring text labels, images, 3D card models, graphs or charts. It’s almost click and drag into the experience that you want. And then you have your thing mark which is a QR-code. That’s the tracking device and everything is planed around it. So you can see the augmented elements certain distances away depending on where you want to be in the experience.

View through the holo-lens. In the left glass you can see the virtual menu that pops up if you scan the QR-code. Foto Reinert.

Do companies need to share their machine data with the vendors of augmented reality?

Stephanie Bolid: Rockwell Automation GmbH uses two different servers right now. We have one server running on a private network. That’s how the holo-lens is connected to the controller. As everyone is using their mobile phone to view the experiences we didn’t want them to be on a private network so we have a second one the public server that is running the mobile experience. Because otherwise everyone could be attached or connected to our controller. You do need something whether it’s public or whether it’s private to collect your data and connect everything. So yes you need to share data because it all needs to be on the same server. That’s your internet of things, it’s the connection of all those things together.