3D Modeling From 2D Images
Lana Madraþeviü, Stjepan Šogoriü
Faculty of Graphic Arts, University of Zagreb
lana.madracevic@irb.hr, stjepan.sogoric@gmail Abstract: This article will give an overview of the methods of
transition from the set of images into 3D model. Direct
method of creating 3D model using 3D software will be
described. Creating p hotorealistic 3D models from a set of
hotograp hs is challenging p roblem in comp uter vision
because the technology is still in its development stage while
the demands for 3D technology and for new methods of 3D
reconstruction are increasing ra
p idly. The traditional
ap roach of comp uter grap hics has been to create a geometric model in 3D and try to rep roduce it onto a two-dimensional image with rendering. A method that is presented in this article, unlike traditional approach, explains ways to create photorealistic 3D models from 2D images. It is mostly focused on detecting, grou ing, and extracting features (edges, faces, etc.) present in a given picture and then trying to interp ret them as three-dimensional clues. The results obtained can be displayed using new technologies such as 3D monitor or p rinted on the 3D p rinter. In op erational p art of this article, it will be sp ecifically exp lained how to create photorealistic human face from the set of images with two different methods automatic and direct. We will offer a solution to the p roblems that occur in the reconstruction of 3D models from photos, such as variations in geometric scale and a mix of textured, uniformly colored, and reflective surfaces.
1.I NTRODUCTION
In the last ten years, a significant progress occurred in the area of 3D graphics. Many studies have been conducted in the field of 3D modeling, and a variety of methods that allow us to reconstruct 2D images into 3D were created. Today, 3D graphics industry creates models that can no longer be disti
nguished from a person in the real world or on photograph. This way of modeling is also the goal of this work; to explore options to create a photorealistic 3D model shaped from the 2D images. This article has listed and briefly described methods of converting 2D images into 3D models. Automatic method of creating and modeling "the head" in F acegen and F aceshop programs and manual processing with 3D modeling programs using photos or film (2D) test with "supek" illustration is described in detail. These methods were chosen because they create a solid 3D model. There are other methods such as stereoscopy, which show us a 3D scene but its depth is illusive. Such methods are good if the ultimate goal is on-screen visualization only, but if you need to create a solid 3D model for 3D printing, it is necessary to use automatic or manual modeling. F urther in the text, the pipeline of modeling method implemented in the practical part is presented.
Pipeline of 3D reconstruction:
Model recording:
qyg
Automatic method of modeling:
-Methods of analysis and synthesis
- Image orientation:
-Alignment
by the angle of recording
Direct manual processing in 3D programs:
- Alignment of images in
Accordance with the movement in 3D space
-------
2.M ODEL RECORDING
This section will describe in detail, the proper way of recording the model with the camera, and automatic method of making the 3D model through the obtained recordings. When it comes to facial reconstruction from series of photographs, there will almost always be some errors that occur due to the moving of the person. One of possible solutions is to simultaneously record with multiple cameras from different angles. Three cameras are distributed in a circle around the subject, angle set to 90° (profiles and in front of the face) simultaneously recording the subject. In such a way we obtain images in which the error of changing the position of the object will be minimal. But in most cases, while recording, there is only one camera, which circles around the subject.
MIPRO 2010, May 24-28, 2010, Opatija, Croatia
Places of recording are shown in Figure 1. The positions of 45° and 135° are used to capture extra profiles.
Figure 1. Shows the camera positions while filming. Positions of
45 ° and 135 ° are used for the control profile.
In addition to errors that occur due to movement of the object and changing the facial expression, more errors occur due to improper circling of the camera around the object, and wrong selection of recording places. They can be solved if a video recording is used instead of photographs. In video recording, we are looking for frames that best match the position of the camera and the person. More shots can be made. It is equally important to obtain images of same illumination, to light the object from every side, to eliminate unwanted shades making the important elements of the face structure not clearly visible, and to observe the change of facial expressions. Most important position while shooting is the frontal position. It serves to provide basic information needed to correct errors from position changing of the subject taken from other angles. If, during the recording, the subject changes position, the basic outline from the frontal picture will be used for correcting shift errors. That is the reason why the frontal image must have a neutral facial expression and position. Illumination is taken as normal and zero. Realistic virtual face contains all the changes that have taken place on a real person face during the filming. Modeling the face we enter all changes and seek a neutral expression, or zero position. That gains us realistic face character. Wrinkles that occu
r on a face while person is smiling are invisible with the serious expression, while in the neutral expression they will be visible regardless of the lips position. This is done by seizing medium values of the position and drawing lines on them. Face animation boosts the key lines.
3.U SE OF HDR METHOD
High Dynamic Range Imaging method and Tone-mapping are largely described in reference [1]. To obtain high quality illumination on image, and to highlight its details hidden in the shade we can use the HDR method. It contains three shots filmed from the same position and embedded in one image. The difference between these shots is the length of exposure. The first shot must be over exposed, second under exposed, and third under optimal exposure. By combining these three shots we obtain the optimal tone values in all areas, with more detail in bright and dark areas. In such a way we increase the quantities of small details, and face acquires specific character. There are two methods of creating HDR images. The real HDR is filmed with three shots as described, while for creating false HDR it requires only one. Method applied in this article is false HDR. We get three shots from one image by changing exposure in specified programs (Adobe Photoshop). When three shots are made, they are merged together into one HDR image. Then, the Tone-mapping method is applied. Obtained images contain lots of detail, strengthened face lines and equal illumination.
4.A UTOMATIC FACE MODELING METHOD Method which automatically creates 3D face model can be obtained with very good results if we use HDR and tone-mapping methods of image processing. This method is good because of speed in creating the solid 3D model, but not good if we want a photorealistic model, since it contains errors such as: the structure of the skin does not exist, and there is only a basic face shape. If the image contains a variety of facial expressions, or lack details because of shadowed parts, the program will take the middle, and will lack the face characterization. The positive side is that this method can be used to create the base model (zero model) for further processing.
4.1.Methods of analysis and synthesis, automatic
reconstruction of 3D models from images
F rom the processed shots, first taken is the frontal face image. It is altered to detect key points of the face. The key points are: the beginning, the middle and the end points of the eyes, lips, nose, chin, forehead, cheeks ... as shown in Figure 2.
Automatic face reconstruction also includes inserting images from which the texture mapping is acquired. It allows you to view details on skin, color, tone and brightness, and it provides nearly photorealistic look. F or this reason it is better to use images obtained from HDR method. Main disadvantage is that without the map applied, the model looks plastic and unreal because there are no physically modeled details on the face. That is why the entire composition of the model and its realistic appearance depends precisely on the quality of images applied on the surface of the model. Another deficiency of such models is the inability to determine the age and characteristic features of the person, because the model is formed by a few given points which are marked on it, while overall
appearance is identical to any model constructed in such a way (shape of the head, ears, hair, and lack of facial detail). The basic form of the head is in the database of the program. The final model contains similarities with the real face (depending on the quality of photos, texture), however, it is far from photorealistic, and requires much additional processing. Such programs are usually based on Alignment algorithm that automatically
ge
aligns the entered picture with the existing model and creates a form based on the given points. Algorithm creates a model that consists of a mesh of points and vectors by taking the data from the photos.
By mapping the texture onto the 3d face geometry, the 3d face model from the input 2d face image is reconstructed.
[2]. Software for automatic reconstruction of 3D models (such as Facegen) provides refinement of the model, which is based on the automatic shifting of face components. It is possible to determine the gender of the created model. The key points of the model can be subsequently adjusted, such as the height of the nose, eyes, lips and forehead, and their mutual relationship (shape of the head, jaw
strength, height and shape of faces, the distance between the eyes, etc.). Manipulation of the texture refers to the highly expressed, which shows all the details of inserted images or texture that is transparent and nearly invisible. The amount of polygons and vertex marks the presence of additional components of the model, such as the eyelids, tongue, form that shows the inside of the mouth and some more additional segments such as several models of hair and glasses. These additional segments may be used during the animation of the model. The program doesn't offer the possibility of animation and rendering, although the models are compatible with programs like 3DS Max or Maya. Model can be exported in various forms that can be recognized by the second 3D program. Obj.file and 3DS.file allow the texture to be exported. Texture contains coordinates that are aligned by the model, i.e. according to the position of vertex and faces. Texture can be subsequently finished in common software for image processing.
4.2.Image orientation
Image orientation involves setting up different profiles depending on the head rotation. If the frontal image is used solely, there will be no texture on the profiles. Blank fields will appear or will be spread on faces, depending on their number and position. There is still no software that would automatically resolve this problem. Different head positions have to contain the exactly same position in reality. Th
e three images here are set to face three positions as follows: frontal shot and the two profiles. The pictures are marked by key points as described. The program automatically applies the selected texture points. We get a good coverage of the face with texture, observed from different angles. During rotation, mesh model has no empty areas, and texture settles on mesh surfaces. Like on Figure 3. Finally, the images that are applied to the model must be fully equal in all characteristics, except by the angle of shooting. After obtaining the 3D model, it is exported as obj.file or 3ds.file, to finish with the direct method.
Figure 3. 3D mash model reconstructed from 2d images of left
profile, right profile and frontal image
5.D IRECT METHOD OF MODELING 3D
OBJECTS FROM 2D IMAGE
Direct method involves manual processing of 3D model,
that is largely supported through CAD programs like 3DS
Max or Maya. Photos that were used to obtain model
automatically, can now be useful for subsequent manual
finishing and stacking texture map for the model. The
advantage of direct methods of modeling is the possibility
of modeling details, rendering the actual display model,
modeling the skin surface and all its details (automated
method only applies the image), ability to increase and
decrease vertex and faces that resizes file, although it
changes details on the mesh, more ways of modeling (poly,
<), the ability to create facial expressions
characteristical for a person without any special photos, the
possibility of animation, rendering, 3D printing and
presenting on 3D monitor by rendering with two cameras.
Disadvantage of direct methods of modeling, programs are
often more complex and not user-friendly, there is no
automation, rich work experience is needed with such
programs in order to obtain a realistic look. Realistic facial
animation is achieved trough geometric and image
manipulations [3]. Geometric distortions and facial
expressions are unique for individual. Manipulation and
photo processing is required in order to obtain reflection
properties of skin, small details and realistic features of
hair and eyes. Obtaining such properties on the model can
be quite complicated, and almost impossible if it is solely
made by geometric manipulation. Here described, is
summarized review of theoretical approaches, and detail
methods of modeling the face.Geometric manipulations
include key-framing and geometric interpolations,
parameterizations, finite element methods, muscle based
modeling, visual simulation using pseudo muscles, spline
models and free-form deformations [3].Image
manipulation includes alignment between multiple images,
texture manipulation, unification of light and shade on the
image, and highlighting the key details that create a
personalized face. Image manipulation includes the
possibility of using HDR methods that will highlight key
lines and details of the face, tone the image, and increase
the amount of detail in brightest or darkest parts, which
would be impossible with the common methods of creating
and processing images. Such treatment also includes tone-
mapping method. In doing so, one has to be careful of the
positions of cameramen and subject, and the face
expression to be neutral. Manual modeling method is
described in [4].Summarized technique of manual
modeling is described as a tutorial that can be used during
the course of manual modeling of the head and face
regardless of the application used. The basic approach of
this modeling is poly modeling with subdivision smoothing
for mash refinement. This way of modeling can be used as
additional processing of existing 3D model obtained by the
automatic method. With direct modeling we acquire the
basic face topology, a mesh that can be easily refined with
existing vertex.
5.1.Alignment of images in accordance with the
movement in 3D space
Crucial for the refinement of 3D models is the existence of photographs that are placed in 3D application space. They are oriented towards the model, so that each side of the model is covered with appropriate image. Background images can be set in two ways. The first way is to set images in a circle around the model at the angles of 45° or 90°. Other way is to place them together, forming a cross, so that they overlap in the middle. The model is located in the middle of the cross.
Figure 5. Shows the orientation of the images toward
the view of the model
Mash of the model must be applied with a texture that has
transparency properties. If the model isn't transparent,
background images will not be visible, and modeling the
object would be impossible. Modeling the object is based
on the key face lines that are visible on the background
cmmi4images.
5.2.Model production
Polygonal network is a set of vertex, edges, faces, forming
a shape of polygonal models. With their merger we get a
closed three-dimensional model. Such models are simple
for design and rendering on the computer. With editable
mash command, it’s easy to compose small details on the
face model. Before we begin modeling, it is necessary to
put the image in 3D space. Mash is created by prominent
edges from the photo, following their shape. Polygons are
moved to coincide with the face in the background of the
mesh. Polygons must be transparent and this also applies
for the entire mesh. The face is shaped so that the vertex
we selected, are aligned with the edges of the photographs
in the background. With "soft-selection" we are able to
automatically select more vertexes or edges at once. A
mesh that is created by moving the vertex follows the form
of the image. It is created from the front view, so edges
move by X and Z coordinates, following the shape of the
eyes, nose, lips, head, etc. For these reasons well processed
images of the model are needed.
When the mash is created in the frontal view, next step is
to create a profile. Transitioning to the side view, the
images are set up in the background so they can overlap
with the model, and the head is modeled in the already
described manner. F ace is mostly symmetrical, it is
necessary to model only one side, while the other side
automatically copies modeling with the mirror command.
Asymmetric details can be addressed later. When modeling
details on the face, we start with key points same as with
the automatic modeling method. We find the key detail
points and formulate a mash, following images in the
background. Model orientates by necessity, but we must be
careful that it is always covered by properly set
background image, which corresponds to the model view.
Mash topology is very important here, because the final
appearance of the model depends on it, just as it enables
the possibility of processing. Mash obtained with the
automatic method contains 76,597 edges and 38,290
vertices which is relatively rough structure of the face
model. It is good for getting a rough mash topology as
shown in fig.6. Polishing the mash can be implemented
through Mash Smooth command, which gets us 302,884
edges and 153,241 vertices. Mash Smooth applies at the
end of the work, so the tiniest details of the surface could
be modeled. Using Mash Smooth for modeling of large
surfaces is difficult.
Figure 6. Using key points to create Mash Topology and using
Mash Smooth.
F ree form deformation (F F D) can deform many types of
surface primitives, including polygons; quadric,
声带小节parametric, and implicit surfaces; and solid models. [3].
This is a way to form the basic mesh of 3D model. It
allows volume deformation of the model. The object is
situated in the middle of a 3D cubic lattice, and is formed
by moving the control points. Shaping of the model is
implemented by combining several types of
F F
D:
Extended free form deformation (EF F D) and Rotational
free form deformation (RFFD). Such procedure allows us a
detailed formation of several face and head structure parts
[3].
Figure 7. Shows the formation of the model using FFD with
background images. Model is surrounded by a control 3D mesh.
By shifting the control points, model within the mesh is
deformed.初中生课外辅导
The next step is a transfer from poly models to NURBS
(Non-uniform rational B-spline). NURBS surface is
defined by cracked curves that depend on the weighted
control points. Curve follows (but not necessarily merges)
points. Changing the properties of these points will change
the curvature of the line. NURBS surfaces are very smooth
because they don’t use the approximation of small flat
surfaces of poly model, and are suitable for organic,
photorealistic modeling. With this we can shape the face of
the model in detail. Surfaces are smooth and there are no
sharp edges from which the face mash is composed.
Subdivision surface is another way of modeling smooth
surfaces. With subdivision surfaces, sharp and rough parts
藏文网are divided into smaller areas, ergo the rough shape of the face gets a smooth formation. The lack of such surface is that it is not sufficiently adapted for the detail construction. With it we can convert only the larger, square areas into a fine curve, while the minor strain, fully align and create a flat surface. For this reason, to create the model of the face, it is necessary to combine the Subdivision surface with NURBS. As a final result, realistic, detailed surface-modeled faces are obtained.
Adding texture to the model is the next step, and although models may look quite detailed, the model without the texture is not very realistic. This process is called Texture Mapping. 2D images are assigned to a particular segment of the 3D model. As the 3D model has XYZ coordinates, 2D texture has UV coordinates, which determine where in the mash surface, the texture will be applied. Texture that is going to be used, are the background images that we used for modeling.
F inally, there is the possibility of animation, because the automated method of modeling created requirements (area below the external faces) for easier animation of the final model.
5.3. Modeling of facial expressions
The next step in modeling is to create facial expressions. It is recommended, that the model doesn’t have a neutral expression, because it doesn’t contain recognizable characteristics of a person. For this reason, the creation of facial expressions on the model should be based on FACS [5]
Table 1. Sample single facial units [3]
Table 2. Example sets of action units for basic expression
FACS explains how to create an expression of feeling on the human face. The Facial Action Coding
System (FACS) is a description of the movements of the facial muscles derived from analysis of facial anatomy. FACS includes 44 basic action units (AUs). Combinations of independent action units generate facial expressions [5, 3]. F or example, to create an expression of smiling faces on the 3D model, for happiness we use AU 1, 6, 12, and 14.
There are many methods of forming the movement of human faces in 3D models. Common feature is that the mash that simulates facial skin must be elastic, and connected with other parts of the face. While modeling smile deformation occurs at the cheeks and eyes.
They should be linked with the complicated structure of muscle and bones when creating realistic animation of "muscular model", but if the static facial expression is modeled, it isn’t necessary to do the subcutaneous structure. In this case only the outer face deformations that occur with such movement can be made, as described in Table 1. Such models are called pseudo muscular models. They are an alternative and simpler approach for creating a strain on the mash entities. Deformation is carried out on the thin shell facial mash. Modeling is done in the same way as other parts of the face, over the polygon mash, free form deformation and non-uniform rational B-spline, as described previously in this article.
6. R ENDERING
The final step of creating a model is rendering. Rendering is a mathematical process of createing 2D images from 3D models. In this case, the rendering will be used only to see the final result of the work. Even though the 3D programs create their own lighting in 3D space, it doesn’t provide a realistic view of the model. Many parts remain shaded and invisible; therefore it is necessary to start the rendering process. It allows us to view the model as it looks in reality. It is required to perform rendering on each side of the model, and that can be achieved in two ways. First way is to turn the view of the model and render each image separately. It provides us with an overview of the model,
and the process is much faster and easier than creating
animation. Other way is to create a short animation of rotating object. This is complicated because, not only the view turns but the model as well. The bright side is that these animations can provide better sight of the object. To get an overview of how the model looks like in reality, it is necessary to set the correct lighting. Light reflection from the object
depends on its outer surface as well as the type of light objects set in 3D space. This process is unavoidable, especially if the ultimate purpose of modeling (except screen viewing), is 3D printing process. Considering that the 3D printing is an expensive process, errors on the model are unaccept
able, so rendering is the only way to find out what will the model look like in reality. It is recommended to execute the process of Photorealistic Rendering if the software grants it. Photorealistic Rendering is a blend of art and science which creates images and scenes that completely mimic the real world.
This can be achieved with a computer algorithm that converts a 3D model to 2D image, while taking the value of realistic illumination. There are several software packages that can be used to achieve this goal. One of the tools that came on the market in order to create these highly sophisticated images is Renderman (Pixar, www.pixar). Renderman is developed and released publicly by Pixar in the 1980's, and it became a standard for 3D photorealistic rendering and animation.
AU F ACS Name AU F
ACS Name 1 Inner Brow Raiser 6 Check Raiser 2 Outer bow raiser 9 Nose wrinkle 4 Brow lower 12 Lid corner puller 5 Upper lid raiser 15 Lid corner depressor 7 Lid tightened 17 Chin raiser 14 dimpler 23 Lip tighter 20 Lip strech 26 Jaw drop Basic expression Involved action units disgust AU2,4,9,15,17 happines AU1,6,12,14 sadness AU1,4,15,23
fear AU1,2,4,5,15,20,26
7.C ONCLUSION
The ultimate goal of current research in the field of modeling is to find a system that creates realistic animation in real-time, is fully automated and easily adapts to the individual’s face. Such a program will probably be available in the near future, as for now, the modeling based on manual and automatic design (which, as seen in testing described in this article), doesn’t provide satisfying results. Ergo, combined procedure was implemented with automatic and direct methods of modeling. Finally, such a procedure has many advantages especially when used with HDR and Tone-mapping methods of processing 2D shots. Some of them are: less time and effort is required to obtain realistic models (because the process is semiautomatic), it’s easier to create, it provides a greater chance of creating realistic models, and finally, getting the texture-mapping arranged by UV coordinates is automatic. It is necessary to record from different angles, and to acquire shots of high resolution and normal illumination, if not the entire process of creating photorealistic model is almost impossible. Realistic look of the model depends solely on the skills and experience of the designer. However, there are many ways to improve the existing method, and to generate space for further research.
A CKNOWLEDGMENT
F irst of all I would like to express my gratitude towards dr.sc. Karolj Skala, professor on Faculty of Graphic Arts, University of Zagreb and head of Center for Informatics and Computing at Ruÿer Boškoviü Institute, for help and support. I am very grateful that I was able to use the Laboratory for Optoelectronics and Visualization, which helped me to develop a desktop design and 3D printing of Academician Ivan Supek pilot busts. At the end I would like to thank my colleague Ana Materni for obtaining material in the form of films and photographs, which were used in the preparation of the model. R EFERENCES
[1] High Dynamic Range Imaging; Greg Ward Exponent – Failure Analysis Assoc. Menlo Park, California
[2] Automatic 3d Reconstruction for face recognition, Yuxiao Hu, Dlong Jiang, Shuicheng Yan, Lei Zhang, Hongjiang Zhang [3] A survey of Facial modeling and animation techniques, Jun-yong Noh and Ulrich Neumann
[4] Model the perfect 3D face, Tutorial, Expertise provided by Simon Danaher.
[5] Facial Action Coding System, P.Ekman, W.V.Friesen, 1978.
豫公网安备41160202000603
站长QQ:729038198
关于我们
投诉建议