I'm going to quickly go over how the threading model has changed in Electric Eye over the last two years.
Originally, Electric Eye was single-threaded. The data flow was:
Thread #1: Get Video Frame → Extract Testable Frame → Get Audio Frame → Run Test Case → Update UI → Repeat
For our original purposes (reducing our range from 300ms to 80ms), this was fine, but it didn't scale to more complex test cases.
Our next threading model split in two: our UI/acquisition thread and our test case thread.
Thread #1: Get Video Frame → Extract Testable Frame → Get Audio Frame → Enqueue Frame → Update UI → Repeat
Thread #2: Dequeue Frame → Run Test Case → Repeat
Thread #2 used a consumer model based on a lock-free queue from C++ Concurrency In Action. However, we started running into issues when we switched over to using UMats in our OpenCV code. Using GPU resources on thread #2 were impacting our UI and causing frame time issues with our acquisition thread.
Our next threading model had us using three threads: acquisition, test case, and UI.
Thread #1: Get Video Frame → Extract Testable Frame → Get Audio Frame → Enqueue Frame → Repeat
Thread #2: Dequeue Frame → Run Test Case → Repeat
Thread #3: Update UI when possible
An early mistake made with this threading model was that we tried to get the video frame into a UMat at the end of thread #1 to speed up thread #2, but this led to us running into resource starvation issues (you can only have so many GPU resources allocated) and it still caused timing issues.
Our final threading model still has three threads, but we shifted where we extracted the frame.
Thread #1: Get Video Frame → Get Audio Frame → Enqueue Frame → Repeat
Thread #2: Dequeue Frame → Extract Testable Frame → Run Test Case → Repeat
Thread #3: Update UI when possible
We are actively ensuring that thread #1 is not using any GPU resources whatsoever. We get our video frame if available (with our exposure controls, this takes us ~3-5ms per frame), grab whatever audio came in during this iteration, generate a FrameData object, and enqueue it up in thread #1.
In thread #2, we extract out the testable frame using code similar to the warpPerspective code I spoke about before with one extra perspective fix and my patented curved screen code, turn the frame into a UMat, run the test case against the extracted frame and/or audio object that we have, and if we have any changes to the UI, we signal the UI thread that it needs to update now.
Thread #3 is just a standard UI thread. It does handle getting commands via IPC from our command-line tool as well, but it just routes them into standard UI commands.
Showing posts with label OpenCV. Show all posts
Showing posts with label OpenCV. Show all posts
February 11, 2018
February 4, 2018
Electric Eye Lite - Introduction
It's still going to be some time before I'm going to be able to do a full source release of Electric Eye through work. Since Electric Eye was revealed to the world, we've done over 100 internal releases and over fifteen new releases to partners. We've dramatically reduced the error bars in our measurements, fixed a lot of bugs, and in general have made a very stable tool. As a result, we have a very stable but very messy codebase.
Over the next month, I'm going to be talking through a clean implementation of the non-patented parts of Electric Eye and walk through the creation of what will essentially be "Electric Eye Lite" or "EEL."
Over the next four posts, I'm going to be talking through each of the three threads inside the codebase (data acquisition, testing, UI), the lessons learned over the last two years of working on the tool, and finally bring it all together in a simple, clean codebase.
All the code will be over on Github licensed under BSD 3-clause.
Talk to you soon.
Over the next month, I'm going to be talking through a clean implementation of the non-patented parts of Electric Eye and walk through the creation of what will essentially be "Electric Eye Lite" or "EEL."
Over the next four posts, I'm going to be talking through each of the three threads inside the codebase (data acquisition, testing, UI), the lessons learned over the last two years of working on the tool, and finally bring it all together in a simple, clean codebase.
All the code will be over on Github licensed under BSD 3-clause.
Talk to you soon.
Tags:
Development,
Electric Eye,
OpenCV
October 26, 2016
Electric Eye Open Source Release Update
I've been asked a few dozen times over the last month about where the open source release of Electric Eye is. Here's a quick update.
I was in Japan and South Korea from October 1 through October 13. Another individual working on Electric Eye was in China October 14 through October 21. We were on site at our partners ensuring that the multicore rewrite that we did over the summer worked the way that it should. A couple of minor bugs surfaced on site, including an interesting one where a S/PDIF renegotiation caused our audio interfaces to no longer receive audio. We're both currently fixing the final spat of issues that surfaced from this trip.
On November 3, we're releasing the code to one partner here in the Bay Area. This is a bit of a trial run to ensure that there's nothing egregiously off about our code.
Once we've gotten final feedback from our local partner about the code, we can resume working with legal to get the code released.
Sorry about the delay. On the upside, we're going to be supporting several more webcam models as part of the release. If you're trying to prepare for the release, I can provide you with system requirements...
The Linux version isn't ready to go yet, and there are two chunks of code that we won't be able to release that won't impact general usage. However, those should not stop the release.
Will share more once I know more.
I was in Japan and South Korea from October 1 through October 13. Another individual working on Electric Eye was in China October 14 through October 21. We were on site at our partners ensuring that the multicore rewrite that we did over the summer worked the way that it should. A couple of minor bugs surfaced on site, including an interesting one where a S/PDIF renegotiation caused our audio interfaces to no longer receive audio. We're both currently fixing the final spat of issues that surfaced from this trip.
On November 3, we're releasing the code to one partner here in the Bay Area. This is a bit of a trial run to ensure that there's nothing egregiously off about our code.
Once we've gotten final feedback from our local partner about the code, we can resume working with legal to get the code released.
Sorry about the delay. On the upside, we're going to be supporting several more webcam models as part of the release. If you're trying to prepare for the release, I can provide you with system requirements...
- Windows
- Windows 7 SP1 64-bit or above with all Windows Updates installed
- Intel i5 2.6 GHz processor or better, or the AMD equivalent
- 8GB of RAM or more
- OpenGL 3.2 compatible video card
- Visual Studio 2015 Update 3
- Mac
- OS X Yosemite or El Capitan (macOS Sierra not yet supported)
- mid-2014 MacBook Pro or better
- Xcode 6.4
- One of the following cameras (affiliate links)
The Linux version isn't ready to go yet, and there are two chunks of code that we won't be able to release that won't impact general usage. However, those should not stop the release.
Will share more once I know more.
Tags:
Electric Eye,
OpenCV
September 22, 2015
Electric Eye - The Post Postmortem
If you haven't read the Netflix tech blog yet today, you might want to wander over there before reading this post. Yep, the project I created and am acting as lead on at Netflix is called Electric Eye.
The project started ~6 months ago as a one-off experiment and has exploded beyond anything I thought could happen because the damned thing kept working even when I thought it wouldn't. It has repeatedly triggered my Impostor Syndrome, making it ramp up to 11 from time to time, because while I had to test computer vision projects before, and had even written a few, I felt that I was at sub-novice level for most of what I've been doing.
I don't want to talk about the project in this post, though. I would like to talk about why this post was written the way that it was.
First, you may be asking why there isn't any code yet. Well, we've been working with legal on the code release. Almost all of the code is going to be open sourced except perhaps one item that I can't talk about yet and some code to hook this into our backend for test reporting. However, that will include my highly optimized frame transformation system, all of the CV and audio DSP helpers I've developed, and possibly ~25 full CV test cases. There's still some refactoring and code cleanup to be done, but I'd expect a code release either late next spring or summer.
Second, you may be wondering why I link to libraries and, specifically, certain functions in the libraries. This is actually a direct result of interactions with a former employer. No, they didn't make me not release code. They showed me that not every employer will let their test teams use open source unless the code is released under a BSD license or something even more lenient. I intentionally wrote the post so that a tester in a restricted environment could find the information necessary to implement a variation on what I'm doing on their own. After all, if I can tell you how to detect a pattern and transform the image to get you a testable frame, that's most of the challenge right there and if you do it on your own, your legal department will have no issues outside of vetting your library licenses. If I give you the code to do it at the same time, then your legal department may freak out.
Finally, you may be wondering why the blog post is out before the code. We're trying to hire some great SDETs and every candidate we've spoken with has been extremely excited by this project. Showing off some cool testing tech is a great way to grow awareness of QA inside Netflix and does wonders to raise interest levels. The sooner we get some great SDETs in here, the better.
We got the final logo (the blue eye) yesterday morning, but I've been pulling screenshots all through the development process. The screenshots in the article range from 1-4 months old.
Well, back to work. Who knows...we might end up doing an AMA somewhere about it.
The project started ~6 months ago as a one-off experiment and has exploded beyond anything I thought could happen because the damned thing kept working even when I thought it wouldn't. It has repeatedly triggered my Impostor Syndrome, making it ramp up to 11 from time to time, because while I had to test computer vision projects before, and had even written a few, I felt that I was at sub-novice level for most of what I've been doing.
I don't want to talk about the project in this post, though. I would like to talk about why this post was written the way that it was.
First, you may be asking why there isn't any code yet. Well, we've been working with legal on the code release. Almost all of the code is going to be open sourced except perhaps one item that I can't talk about yet and some code to hook this into our backend for test reporting. However, that will include my highly optimized frame transformation system, all of the CV and audio DSP helpers I've developed, and possibly ~25 full CV test cases. There's still some refactoring and code cleanup to be done, but I'd expect a code release either late next spring or summer.
Second, you may be wondering why I link to libraries and, specifically, certain functions in the libraries. This is actually a direct result of interactions with a former employer. No, they didn't make me not release code. They showed me that not every employer will let their test teams use open source unless the code is released under a BSD license or something even more lenient. I intentionally wrote the post so that a tester in a restricted environment could find the information necessary to implement a variation on what I'm doing on their own. After all, if I can tell you how to detect a pattern and transform the image to get you a testable frame, that's most of the challenge right there and if you do it on your own, your legal department will have no issues outside of vetting your library licenses. If I give you the code to do it at the same time, then your legal department may freak out.
Finally, you may be wondering why the blog post is out before the code. We're trying to hire some great SDETs and every candidate we've spoken with has been extremely excited by this project. Showing off some cool testing tech is a great way to grow awareness of QA inside Netflix and does wonders to raise interest levels. The sooner we get some great SDETs in here, the better.
We got the final logo (the blue eye) yesterday morning, but I've been pulling screenshots all through the development process. The screenshots in the article range from 1-4 months old.
Well, back to work. Who knows...we might end up doing an AMA somewhere about it.
Tags:
Development,
Electric Eye,
Netflix,
OpenCV,
Testing
July 6, 2015
Calculate OpenCV warpPerspective Map For Reuse (C++)
At work, I was working on an OpenCV project that utilized warpPerspective quite extensively for real-time perspective correction. It worked great, but it was slow. When I profiled my solution, warpPerspective was taking ~20% of the CPU on my MacBook Pro.
Now, if you look at the code for warpPerspective, it's essentially doing quite a bit of matrix multiplication for each pixel. Since I'm assuming that my camera won't change position during a run, that's a lot of wasted effort.
I use the following code to calculate the map that is generated by warpPerspective up front so I can use it in a remap call later to get the same results for less than half the CPU cost. I store the final map in cv::Mats named transformation_x and transformation_y.
transformationMatrix = cv::getPerspectiveTransform(originalCorners, destinationCorners);
// Since the camera won't be moving, let's pregenerate the remap LUT
cv::Mat inverseTransMatrix;
cv::invert(transformationMatrix, inverseTransMatrix);
// Generate the warp matrix
cv::Mat map_x, map_y, srcTM;
srcTM = inverseTransMatrix.clone(); // If WARP_INVERSE, set srcTM to transformationMatrix
map_x.create(sourceFrame.size(), CV_32FC1);
map_y.create(sourceFrame.size(), CV_32FC1);
double M11, M12, M13, M21, M22, M23, M31, M32, M33;
M11 = srcTM.at<double>(0,0);
M12 = srcTM.at<double>(0,1);
M13 = srcTM.at<double>(0,2);
M21 = srcTM.at<double>(1,0);
M22 = srcTM.at<double>(1,1);
M23 = srcTM.at<double>(1,2);
M31 = srcTM.at<double>(2,0);
M32 = srcTM.at<double>(2,1);
M33 = srcTM.at<double>(2,2);
for (int y = 0; y < sourceFrame.rows; y++) {
double fy = (double)y;
for (int x = 0; x < sourceFrame.cols; x++) {
double fx = (double)x;
double w = ((M31 * fx) + (M32 * fy) + M33);
w = w != 0.0f ? 1.f / w : 0.0f;
float new_x = (float)((M11 * fx) + (M12 * fy) + M13) * w;
float new_y = (float)((M21 * fx) + (M22 * fy) + M23) * w;
map_x.at<float>(y,x) = new_x;
map_y.at<float>(y,x) = new_y;
}
}
// This creates a fixed-point representation of the mapping resulting in ~4% CPU savings
transformation_x.create(sourceFrame.size(), CV_16SC2);
transformation_y.create(sourceFrame.size(), CV_16UC1);
cv::convertMaps(map_x, map_y, transformation_x, transformation_y, false);
// If the fixed-point representation causes issues, replace it with this code
//transformation_x = map_x.clone();
//transformation_y = map_y.clone();
I then apply the map using a remap call:
cv::remap(sourceImage, destinationImage, transformation_x, transformation_y, CV_INTER_LINEAR);
Ended up dropping CPU usage for the call down to ~8%. I'll take it.
Now, if you look at the code for warpPerspective, it's essentially doing quite a bit of matrix multiplication for each pixel. Since I'm assuming that my camera won't change position during a run, that's a lot of wasted effort.
I use the following code to calculate the map that is generated by warpPerspective up front so I can use it in a remap call later to get the same results for less than half the CPU cost. I store the final map in cv::Mats named transformation_x and transformation_y.
transformationMatrix = cv::getPerspectiveTransform(originalCorners, destinationCorners);
// Since the camera won't be moving, let's pregenerate the remap LUT
cv::Mat inverseTransMatrix;
cv::invert(transformationMatrix, inverseTransMatrix);
// Generate the warp matrix
cv::Mat map_x, map_y, srcTM;
srcTM = inverseTransMatrix.clone(); // If WARP_INVERSE, set srcTM to transformationMatrix
map_x.create(sourceFrame.size(), CV_32FC1);
map_y.create(sourceFrame.size(), CV_32FC1);
double M11, M12, M13, M21, M22, M23, M31, M32, M33;
M11 = srcTM.at<double>(0,0);
M12 = srcTM.at<double>(0,1);
M13 = srcTM.at<double>(0,2);
M21 = srcTM.at<double>(1,0);
M22 = srcTM.at<double>(1,1);
M23 = srcTM.at<double>(1,2);
M31 = srcTM.at<double>(2,0);
M32 = srcTM.at<double>(2,1);
M33 = srcTM.at<double>(2,2);
for (int y = 0; y < sourceFrame.rows; y++) {
double fy = (double)y;
for (int x = 0; x < sourceFrame.cols; x++) {
double fx = (double)x;
double w = ((M31 * fx) + (M32 * fy) + M33);
w = w != 0.0f ? 1.f / w : 0.0f;
float new_x = (float)((M11 * fx) + (M12 * fy) + M13) * w;
float new_y = (float)((M21 * fx) + (M22 * fy) + M23) * w;
map_x.at<float>(y,x) = new_x;
map_y.at<float>(y,x) = new_y;
}
}
// This creates a fixed-point representation of the mapping resulting in ~4% CPU savings
transformation_x.create(sourceFrame.size(), CV_16SC2);
transformation_y.create(sourceFrame.size(), CV_16UC1);
cv::convertMaps(map_x, map_y, transformation_x, transformation_y, false);
// If the fixed-point representation causes issues, replace it with this code
//transformation_x = map_x.clone();
//transformation_y = map_y.clone();
I then apply the map using a remap call:
cv::remap(sourceImage, destinationImage, transformation_x, transformation_y, CV_INTER_LINEAR);
Ended up dropping CPU usage for the call down to ~8%. I'll take it.
Tags:
Development,
OpenCV
Subscribe to:
Posts (Atom)