Champion project of Mercari Hackathon 2018
Champion project of Mercari Hackathon 2018
https://b-z.github.io/naruhodo
Naruhodo! means “I see!” or “Aha moment” in Japanese.
This application is designed for students, using AR technology to help them do some wonderful scientific experiments at home.
Currently, it provides a simple optical experiment for demonstration. Users are free to use lenses, mirrors, etc. in it.
Education is the foundation of a country. Equal and quality education is very important. The imbalance of educational resources is a common problem between China and Japan. In a top school, students receive the best education and can do research in the laboratory, which is not possible for students in other schools.
I was born in a poor city. In my student days, I enjoyed to do interesting scientific experiments at home. For example, give me a copper wire (Cu), I will heat it on the fire to get copper oxide (CuO), then put the copper oxide into vinegar (CH~3~COOH), I will finally get a beautiful blue copper acetate (Cu(CH~3~COO)~2~) solution. But a serious problem is that my home cannot always provide the right experiment tools. And it is not safe to do some experiments at home.
AR (Augmented Reality) is a new technology which can mix the real world and the virtual world. I hope to use this technology to help students who are interested in science, to do interesting experiments at home.
First, you should print some markers (naruhodo/markers-print/print.pdf):
The last two together constitute a light source, and the other four markers are convex lens, concave lens, spherical mirror and plane mirror, respectively.
Place these markers on your desk and play!
Currently, the core of Naruhodo! is designed as a three-layer structure:
AR.js analyzes the position and orientation of the markers relative to the camera from the video stream.three.js to render the calculated models on the web page.Here, Ar.js and three.js are two famous open-source libraries. They provide basic AR and model rendering capabilities, respectively. Since they only provide very basic APIs, I still have a lot of work to do. For example, to make the detection more robust in Layer-3, and to generate 3D models and light sources in Layer-1.
Naruhodo! should be visited through an https website, otherwise it has no permission to open the camera of your iOS device.
To set-up a simple https server, you can do this:
npm install -g http-serveropenssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pemhttp-server -S -C cert.pem -o -c-1
Then, open https://localhost:8080.
My inspiration comes from Nintendo LABO and teamLab
‘s exhibitions. I think an educational application should not be limited to virtual, but should be combined with the reality.
 |
 |
|---|---|
| Nintendo LABO | teamLab |
Test the detection of the markers
Add a plane according to the average position & rotation of the markers (deprecated now)
Add a light to the scene
Draw a thick laser between two points
A point light 
Add GUI to the prototype. (Mobile & Computer)
Emmmmm…An invisible concave mirror.
The concave mirror is shown both on mobile and computer
A invisible convex lens
The concave mirror is now visible
Some notes
Draw a lemon 
(误
Test the refraction (deprecated now)
Test complex cases
Add a texture to the mirror (deprecated now) and support colorful lasers
Add some textures to the elements
Use average rotation and height among the markers
Draw a base under each element
Turn markers into bricks
Draw shadows
Draw a logo under the light source
The final version
The presentation
* [x] 插值光源位置* [ ] ~~插值其他器件~~
* 是因为光线从玻璃射出的时候发生了全反射，导致折射公式求解失败
* [ ] ~~是否要做渐变的光线? 比如opacity越来越高~~
和凸透镜、凹透镜、球面镜求交均可以看成和球面镜求交
分两步:
假设光线由$(x_0, y_0, z_0)$射出，方向为$(a, b, c)$，那么光线可以表示为
(x_0, y_0, z_0) + t(a, b, c), t>\varepsilon，
假设球是$(x-x_b)^2+(y-y_b)^2+(z-z_b)^2=R^2$，那么相交时有
(x_0+ta-x_b)^2+(y_0+tb-y_b)^2+(z_0+tc-z_b)^2=R^2，
得到关于$t$的二次方程:
(a^2+b^2+c^2)t^2+\2(a(x_0-x_b)+b(y_0-y_b)+c(z_0-z_b))t+\((x_0-x_b)^2+(y_0-y_b)^2+(z_0-z_b)^2-R^2)=0
可由此解出$t$，并进一步得到交点。判断交点合法性可以求交点到镜面中心的距离是否在合法范围。
function testIntersectionToSpherePart(src, dir, center, R, r, e_center) {var p = testIntersectionToSphere(src, dir, center, R);for (var q of p) {var d = q.distanceTo(e_center);var t = Math.sqrt(sqr(r) + sqr(R - Math.sqrt(sqr(R) - sqr(r))));if (d <= t) {var norm = center.clone().sub(q).normalize();return {pos: q,norm: norm};}}return null;}function testIntersectionToSphere(src, dir, center, R) {var a = dir.dot(dir);var sub = src.clone().sub(center);var b = 2 * dir.dot(sub);var c = sub.dot(sub) - R * R;var delta = b * b - 4 * a * c;var result = [];if (delta >= 0) {delta = Math.sqrt(delta);var t1 = (-b - delta) / a / 2;var t2 = (-b + delta) / a / 2;if (t1 > epsilon) result.push(src.clone().add(dir.clone().multiplyScalar(t1)));if (t2 > epsilon) result.push(src.clone().add(dir.clone().multiplyScalar(t2)));}return result;}
和平面镜求交
平面镜的空间表示是一个中心点c加上x、y方向两个正交向量的线性组合，也就是说，
src+t\cdot dir=c+a\cdot x+b\cdot y
满足$-1\le a\le 1$，$-1\le b\le 1$，$t>\varepsilon$。
其中$a$、$b$、$t$是标量的未知数，其他向量可以挪到等号一边，变成三元一次方程，对三阶矩阵求逆即可。
function testIntersectionToPlanePart(src, dir, c, x, y, norm) {var a1 = x.x; var a2 = x.y; var a3 = x.z;var b1 = y.x; var b2 = y.y; var b3 = y.z;var c1 = -dir.x; var c2 = -dir.y; var c3 = -dir.z;var det = a1 * (b2 * c3 - c2 * b3) - a2 * (b1 * c3 - c1 * b3) + a3 * (b1 * c2 - c1 * b2);if (Math.abs(det) < epsilon) return null;var pa = new THREE.Vector3(b2 * c3 - c2 * b3, c1 * b3 - b1 * c3, b1 * c2 - c1 * b2);var pb = new THREE.Vector3(c2 * a3 - a2 * c3, a1 * c3 - c1 * a3, c1 * a2 - a1 * c2);var pt = new THREE.Vector3(a2 * b3 - b2 * a3, b1 * a3 - a1 * b3, a1 * b2 - b1 * a2);var s = src.clone().sub(c);var a = pa.dot(s) / det;var b = pb.dot(s) / det;var t = pt.dot(s) / det;if (t < epsilon) return null;if (a > 1 || b > 1 || a < -1 || b < -1) return null;var pos = c.clone().add(x.clone().multiplyScalar(a)).add(y.clone().multiplyScalar(b));return {pos: pos,norm: norm}}
首先对所有元件求交，在所有交点中取最近的。
然后针对不同元件做进一步处理:
如果经过元件变换进一步生成了新的光线，那么继续投射这根新的光线。
看起来简单但实际非常复杂。。
朝向均衡
要让所有元件的坐标系的y轴方向统一。
尝试了多种方案，最后可行的方法是，求出每个元件坐标系下(0, 1, 0)的世界坐标v[i]，求它们的均值dir，归一化。
对于每个元件，用v[i]叉乘dir得到旋转轴，点乘得到旋转角，再绕着这个轴转即可。
位置均衡
朝向均衡之后，各个元件坐标系的y轴变得平行。位置均衡可以确保各个元件的$xOz$平面在同一平面上。
首先在第一个元件的坐标系下，表示出(1, 0, 0) (0, 1, 0) (0, 0, 1)三个正交单位向量:
var x = new THREE.Vector3(1, 0, 0);var y = new THREE.Vector3(0, 1, 0);var z = new THREE.Vector3(0, 0, 1);x.applyMatrix4(m[0].matrixWorld).sub(m[0].position);y.applyMatrix4(m[0].matrixWorld).sub(m[0].position);z.applyMatrix4(m[0].matrixWorld).sub(m[0].position);
然后这三个向量写成列向量再并起来所构成的方阵，右边乘上列向量$(a, b, c)^T$能得到在一号元件坐标系下，坐标(a, b, c)在世界坐标系中的表示。假设这个世界坐标系的坐标是(p, q, r)，那么可以在这个坐标左边乘上上面那个方阵的逆矩阵，就得到了在一号元件坐标系下的坐标。
现在，我们用这种方法，把所有元件的坐标系原点放在一号元件坐标系下表示。然后求它们的y坐标的均值，用y的均值更新原有的y，就得到了均衡之后的坐标。
需要加一个能cast出阴影的光源，可以是点光源或者平行光源。对性能要求较高。
每个阴影光源相当于一个相机，需要设置各种相机参数，以及shadowmap的分辨率。分辨率、相机可视范围、软件性能需要做一个权衡。
这里使用的设计是阴影光源固定在场景中，使用平行光源，投射方向也是固定的。根据marker所在平面位置，生成一个接收影子的透明材质，可以比较真实地模拟桌面。