It’s easy to skip past anything that talks about artificial intelligence, if only because it’s so often misused to promote things that can be fairly described as artificial but probably aren’t very intelligent. All of the references to AI in this article should be read with that in mind. Even so, when Sony tells us that it has integrated AI into the same single-chip device as an imaging sensor in its IMX500 series, well, that’s interesting from all sorts of angles.
The company has been manufacturing imaging sensors with the IMX prefix for some time, and it’ll be a familiar range of products to anyone who’s looked into some of the indie camera projects which have popped up in the last few years. Most of them are mainly intended for machine and computer vision applications, so it’s no surprise to find that the press release talks about using the new AI provision on the Sony IMX500 series to do things like identify areas that are particularly popular with shoppers in retail premises (the term is apparently “heat mapping,” though it has nothing directly to do with heat). One of the examples shows the sensor looking at a dog, and outputting, instead of pixel values describing the image, the word “dog.”
This makes a lot of sense for applications where you aren’t really interested in the specifics of the image, but what’s going on; it’s certainly a more efficient way to handle the data. The device is reprogrammable and, depending on the ingenuity of the software engineer, could be used to do more than identifying dogs.
Wouldn’t it be nice to have a camera that could read the timecode from a smart slate? That probably already exists as a plugin for some piece of software, but Sony’s new toy could do it right on the chip. In some ways it’s reminiscent of the human eye, which does some edge detection right on the retina; it’s not just detecting light values, but actually understanding to some extent what it’s seeing.
Why place AI on the chip?
The possibilities of this for film and TV are really limited only by the imagination, though at the most basic level, putting an AI core on the imaging sensor doesn’t give us anything, technologically, that couldn’t have been done by putting the same capability right alongside the sensor. It might improve power consumption a little by reducing the need for complex, high-speed data buses running around the camera, which would be no bad thing. It would also make things easier for manufacturers who need a compact solution. The GoPro and drone cameras of the world might do well, and more integration tends to make things considerably cheaper too.
What’s perhaps more interesting is the wider applicability of AI in things like noise reduction and compression, both of which are already things that happen in cameras. Noise reduction, in particular, has been a particularly important part of improving dynamic range in modern cameras, and if AI could make it work better, with fewer artefacts and for less power consumption, that might be great. Codecs – well – nobody’s going to argue with the idea of going home earlier because the files are smaller while still looking good, at least if anyone can come up with a way of measuring how good a codec is that everyone agrees is valid.
Perhaps we could apply AI to that, too.