Summary: Recreating textured oil painting using brushstroke primitives
Team Members: Aayush Gupta, Ethan Yu, William John Molnar-Brock, Yulia Nugroho
The purpose of this project is to create a realistic oil painting and its brushstroke texture using primitives. The reconstruction algorithm will approximate the initial photo but using exclusively our preset primitives, which will consist of a combination of 2D shapes and 3D brush strokes that will be made using 3D modeling software.
AI image generation models are rapidly increasing in use due to their speed, ease of use, and beauty, democratizing art to be accessible to even those without an artistic background. However, there still exist limitations in AI-image generation: because they mostly work at a pixel-level, and they do not have the texture which is a valuable quality in manmade paintings. We propose to solve these problems by modifying the image generation process by developing a primitive-based reconstruction algorithm that will approximate the initial photo but using exclusively our preset primitives. We would use brush strokes as primitives to ensure that the resulting art looks authentic, mimicking what is accomplished by the human hand. Additionally, for each of our brushstroke primitives, we will have a 3D mesh representing the height map of paint on the canvas that would result from such a brushstroke. This will allow us to generate a composition 3D mesh of the canvas with vertices that reflect the texture of the paint. Although beyond the scope of our graphics-based project, this lays the foundation for 3D printing of the canvas mesh, and then superimposing the 2D primitive-based image over it using rough-surface inking techniques. Although intended to be used for AI-generated images, this approach could be also used for adding a rustic textural element to the printing of sentimental photos.
Milestone 1, which is inspired by the following Primitive project is our baseline plan, with a modification: the original project used geometric primitives like squares and circles, whereas we plan to instead use realistic brushstrokes as primitives.
Milestones 2 and 3 represent the more ambitious and original aspects of our project and thus comprise our aspirational plan as they rely on a strong working version of Milestone 1.
Milestone 1:
Our first deliverable is to make a program that will convert a 2D image into another 2D image of the same dimension that is populated entirely by primitives. To do this we will need to accomplish the following subgoals:
Making a diverse library of 2D brushstrokes to be used as the primitives, and possibly other shapes intended to represent the effect of physical art supplies if time allows (e.g. pen and ink or pastel dust). Although this approach has been shown to be efficient using simple primitives, we may run into difficulty using that approach for more sophisticated primitives. To overcome this issue, we are prepared to redesign the primitive placement algorithm, such as placing the brushstroke based on a lower resolution representation of it (e.g. a gaussian color splash approximating it) to make it more amenable to optimized placement.
Making an efficient algorithm that starts with a blank canvas and progressively adds in rescaled and colored primitives that minimize the difference between the current image and our original input image. This will be done starting from scratch.
Milestone 2:
Develop 3D meshes associated with each of the 2D brushstrokes using the 3d modeling software Houdini or similar.
Develop an efficient method for combining each of the rescaled 3D meshes as produced in milestone 1.2 into a merged mesh that would represent the textured mesh of the canvas surface.
Render the resulting 3D canvas mesh with interpolation of the colors corresponding to each brushstroke and we will arrange each color with different reflectance(This will require some modification of our existing rendering project from earlier in this class to capture the effect of overlapping strokes of paint to give added realism). These renderings will be shown from multiple extreme angles to demonstrate the textured nature of the canvas.
Milestone 3:
Our third milestone will be based on demonstrating whether our approach has improved the beauty of the artworks. To accomplish this, we will ask a group of human to provide a numerical evaluation of how beautiful they perceive the original compared to the modified images, and then present a graph showing how the ratings compared between these two groups.
Measuring the Quality/Performance
While the primitive-based reconstruction algorithm will give us a quantitative score of the quality of reconstruction, we will primarily measure the quality of our algorithm qualitatively, based on whether or not the created images are visually appealing while matching the content of the original image. This will be done by the aforementioned human evaluation of the images produced by our algorithm.
We hope to answer the following questions
Do primitive-based brushstrokes provide more compelling images compared to pixel-level generation?
Does having a component of texture in an image make it more aesthetically pleasing? Can this be implemented efficiently?
Week 1: Start developing the brushstrokes (Milestone 1.1) and begin the development of the primitive-based reconstruction algorithm (Milestone 1.2)
Week 2: Finish Milestone 1 and begin Milestone 2 (incorporating 3D brushstrokes).
Week 3: Finish Milestone 2 and generated the extreme renderings of the textured stylized canvases, and generated comparison (flat) renderings from the same perspectives for points of comparison.
Week 4: Finalize earlier parts and produce a set of stylized images associated with their input images to be used for Milestone 3's aesthetic comparison survey. Writeup report and construct final website to present results.
Github repository: https://github.com/fogleman/primitive