Nekochu commited on
Commit
5818455
·
verified ·
1 Parent(s): 1b10841
Files changed (4) hide show
  1. README.md +83 -13
  2. app.py +618 -0
  3. requirements.txt +32 -0
  4. shapes/bunny.obj +0 -0
README.md CHANGED
@@ -1,13 +1,83 @@
1
- ---
2
- title: TEXTurePaper Broke
3
- emoji: 🦀
4
- colorFrom: yellow
5
- colorTo: red
6
- sdk: gradio
7
- sdk_version: 6.2.0
8
- app_file: app.py
9
- pinned: false
10
- short_description: 'text-to-UV-texture '
11
- ---
12
-
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: TEXTure CPU Lite
3
+ emoji: 🎨
4
+ colorFrom: purple
5
+ colorTo: blue
6
+ sdk: gradio
7
+ sdk_version: 6.2.0
8
+ app_file: app.py
9
+ pinned: false
10
+ license: mit
11
+ ---
12
+
13
+ # TEXTure CPU Lite - Text-Guided 3D Texturing
14
+
15
+ Generate **proper UV texture maps** for 3D meshes using text prompts.
16
+ Uses SD-2-Depth (same as original TEXTure paper) for depth-conditioned generation and xatlas for UV unwrapping.
17
+
18
+ ## Features
19
+
20
+ - **Proper UV Texture Output**: Creates UV atlas that can be applied to the mesh
21
+ - **SD-2-Depth**: Native depth conditioning (same model as original TEXTure)
22
+ - **OBJ Export**: Outputs OBJ + MTL + texture PNG ready to use
23
+ - **xatlas UV Unwrapping**: Automatic UV coordinate generation
24
+
25
+ ## How it works
26
+
27
+ 1. Upload a 3D mesh (.obj, .stl, .ply, .glb)
28
+ 2. Enter a text prompt describing the desired texture
29
+ 3. xatlas generates UV coordinates
30
+ 4. Depth is rendered from multiple viewpoints
31
+ 5. SD-2-Depth generates textures conditioned on depth
32
+ 6. Textures are projected back to UV space
33
+ 7. Download textured mesh (OBJ + MTL + PNG)
34
+
35
+ ## Models Used
36
+
37
+ | Component | Model | Size |
38
+ |-----------|-------|------|
39
+ | SD-2-Depth | radames/stable-diffusion-2-depth-img2img | ~5GB |
40
+
41
+ **Same architecture as original TEXTure paper** (SD-2-Depth) - has native depth conditioning built-in.
42
+ Uses public community copy since official `stabilityai/stable-diffusion-2-depth` is gated.
43
+
44
+ **Runtime:**
45
+ - First run: Downloads ~5GB model (cached in `~/.cache/huggingface/`)
46
+ - CPU uses INT8 quantization via `optimum.quanto` for 3-5x speedup
47
+ - ONNX not supported (SD-2-Depth has 5-channel UNet)
48
+
49
+ **Debug logs:** Check build logs for `[OK]`, `[ERROR]`, `[WARN]` messages if something fails.
50
+
51
+ ## Performance
52
+
53
+ | Device | Steps | Time per View | 4 Views Total |
54
+ |--------|-------|---------------|---------------|
55
+ | GPU (CUDA) | 20 | ~2-3 sec | ~12 sec |
56
+ | CPU (INT8) | 10 | ~1.5 min | ~6 min |
57
+ | CPU (INT8) | 20 | ~3 min | ~14 min |
58
+
59
+ **Tip:** For faster results on CPU, use fewer steps (5-10) or fewer views (2-3).
60
+
61
+ ## Local Development
62
+
63
+ ```bash
64
+ pip install -r requirements.txt
65
+ python app.py
66
+ ```
67
+
68
+ ## Files Structure
69
+
70
+ ```
71
+ ├── app.py # Single-file implementation
72
+ ├── requirements.txt
73
+ ├── README.md
74
+ └── shapes/
75
+ └── bunny.obj # Sample mesh
76
+ ```
77
+
78
+ ## Credits
79
+
80
+ - [TEXTure Paper](https://texturepaper.github.io/TEXTurePaper/) - Yael Vinker et al.
81
+ - [ControlNet](https://github.com/lllyasviel/ControlNet) - Lvmin Zhang
82
+ - [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) - RunwayML
83
+ - [xatlas](https://github.com/jpcy/xatlas) - UV unwrapping
app.py ADDED
@@ -0,0 +1,618 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ TEXTure CPU Lite - Text-Guided 3D Texturing
3
+ Single-file implementation with CPU renderer and xatlas UV unwrapping.
4
+ """
5
+ import os
6
+ import copy
7
+ import tempfile
8
+ import shutil
9
+ import zipfile
10
+ import numpy as np
11
+ import torch
12
+ import torch.nn as nn
13
+ import torch.nn.functional as F
14
+ import trimesh
15
+ import gradio as gr
16
+ from PIL import Image
17
+ from pathlib import Path
18
+ from typing import Optional, Dict, Any, Tuple
19
+ from dataclasses import dataclass
20
+
21
+ # =============================================================================
22
+ # CONFIGURATION
23
+ # =============================================================================
24
+ SD_MODEL = "radames/stable-diffusion-2-depth-img2img" # Public copy of SD-2-Depth (original is gated)
25
+ DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
26
+ DTYPE = torch.float16 if DEVICE == "cuda" else torch.float32
27
+ NUM_VIEWS = 4
28
+ RENDER_SIZE = 512
29
+ TEXTURE_RESOLUTION = 1024
30
+ NUM_INFERENCE_STEPS = 20 # Full quality (slower on CPU but better results)
31
+
32
+ # =============================================================================
33
+ # MESH CLASS (replaces Kaolin)
34
+ # =============================================================================
35
+ class Mesh:
36
+ """CPU-compatible mesh class using trimesh."""
37
+
38
+ def __init__(self, obj_path: str, device: str = "cpu"):
39
+ mesh = trimesh.load(obj_path, force='mesh', process=False)
40
+ if not isinstance(mesh, trimesh.Trimesh):
41
+ raise ValueError(f"Failed to load mesh from {obj_path}")
42
+
43
+ self.vertices = torch.tensor(mesh.vertices, dtype=torch.float32, device=device)
44
+ self.faces = torch.tensor(mesh.faces, dtype=torch.long, device=device)
45
+ self.normals, self.face_area = self._calc_normals(self.vertices, self.faces)
46
+ self.ft = None
47
+ self.vt = None
48
+
49
+ if hasattr(mesh.visual, 'uv') and mesh.visual.uv is not None:
50
+ uv = mesh.visual.uv
51
+ if uv is not None and len(uv) > 0:
52
+ self.vt = torch.tensor(uv, dtype=torch.float32, device=device)
53
+ self.ft = self.faces.clone()
54
+
55
+ @staticmethod
56
+ def _calc_normals(vertices, faces):
57
+ v0, v1, v2 = vertices[faces[:, 0]], vertices[faces[:, 1]], vertices[faces[:, 2]]
58
+ n = torch.cross(v1 - v0, v2 - v0, dim=-1)
59
+ area = torch.norm(n, dim=-1)
60
+ n = n / (area[:, None] + 1e-8)
61
+ return n, area / 2
62
+
63
+ def normalize_mesh(self, inplace=False, target_scale=1.0, dy=0.0):
64
+ mesh = self if inplace else copy.deepcopy(self)
65
+ verts = mesh.vertices
66
+ center = verts.mean(dim=0)
67
+ verts = verts - center
68
+ scale = torch.max(torch.norm(verts, p=2, dim=1))
69
+ verts = verts / (scale + 1e-8) * target_scale
70
+ verts[:, 1] = verts[:, 1] + dy
71
+ mesh.vertices = verts
72
+ return mesh
73
+
74
+ # =============================================================================
75
+ # RENDERER (replaces Kaolin render functions)
76
+ # =============================================================================
77
+ def perspective_projection(fov=np.pi/3, aspect=1.0, near=0.1, far=100.0):
78
+ f = 1.0 / np.tan(fov / 2)
79
+ proj = torch.zeros(4, 4)
80
+ proj[0, 0] = f / aspect
81
+ proj[1, 1] = f
82
+ proj[2, 2] = (far + near) / (near - far)
83
+ proj[2, 3] = (2 * far * near) / (near - far)
84
+ proj[3, 2] = -1.0
85
+ return proj
86
+
87
+ def view_matrix(pos, look_at, up):
88
+ pos, look_at, up = pos.squeeze(), look_at.squeeze(), up.squeeze()
89
+ forward = (look_at - pos) / (torch.norm(look_at - pos) + 1e-8)
90
+ right = torch.linalg.cross(forward, up)
91
+ right = right / (torch.norm(right) + 1e-8)
92
+ new_up = torch.linalg.cross(right, forward)
93
+ view = torch.eye(4)
94
+ view[0, :3], view[1, :3], view[2, :3] = right, new_up, -forward
95
+ view[0, 3] = -torch.dot(right, pos)
96
+ view[1, 3] = -torch.dot(new_up, pos)
97
+ view[2, 3] = torch.dot(forward, pos)
98
+ return view.unsqueeze(0)
99
+
100
+ def camera_from_angles(elev, azim, r=3.0, look_at_height=0.0):
101
+ x = r * torch.sin(elev) * torch.sin(azim)
102
+ y = r * torch.cos(elev)
103
+ z = r * torch.sin(elev) * torch.cos(azim)
104
+ pos = torch.tensor([[x, y, z]])
105
+ look_at = torch.zeros_like(pos)
106
+ look_at[:, 1] = look_at_height
107
+ return view_matrix(pos, look_at, torch.tensor([[0.0, 1.0, 0.0]]))
108
+
109
+ def prepare_vertices(vertices, faces, proj, view):
110
+ device = vertices.device
111
+ face_verts = vertices[faces.long()]
112
+ ones = torch.ones(*face_verts.shape[:-1], 1, device=device)
113
+ face_verts_h = torch.cat([face_verts, ones], dim=-1)
114
+
115
+ view_mat = view.squeeze(0).to(device)
116
+ face_verts_cam = torch.einsum('ij,fvj->fvi', view_mat, face_verts_h)
117
+
118
+ proj_mat = proj.to(device)
119
+ face_verts_clip = torch.einsum('ij,fvj->fvi', proj_mat, face_verts_cam)
120
+
121
+ w = face_verts_clip[..., 3:4].clamp(min=1e-8)
122
+ face_verts_ndc = face_verts_clip[..., :3] / w
123
+ face_verts_img = face_verts_ndc[..., :2]
124
+
125
+ v0, v1, v2 = face_verts[:, 0], face_verts[:, 1], face_verts[:, 2]
126
+ normals = torch.cross(v1 - v0, v2 - v0, dim=-1)
127
+ normals = normals / (torch.norm(normals, dim=-1, keepdim=True) + 1e-8)
128
+
129
+ return face_verts_cam.unsqueeze(0), face_verts_img.unsqueeze(0), normals.unsqueeze(0)
130
+
131
+ def rasterize(width, height, face_z, face_verts_img, face_attrs):
132
+ device = face_verts_img.device
133
+ num_faces = face_verts_img.shape[1]
134
+ num_attrs = face_attrs.shape[-1]
135
+
136
+ features = torch.zeros(1, height, width, num_attrs, device=device)
137
+ face_idx = torch.full((1, height, width, 1), -1, dtype=torch.long, device=device)
138
+ depth_buf = torch.full((1, height, width), float('inf'), device=device)
139
+
140
+ verts_pix = face_verts_img.clone()
141
+ verts_pix[..., 0] = (verts_pix[..., 0] + 1) * 0.5 * width
142
+ verts_pix[..., 1] = (1 - verts_pix[..., 1]) * 0.5 * height
143
+
144
+ for f in range(num_faces):
145
+ v0, v1, v2 = verts_pix[0, f, 0], verts_pix[0, f, 1], verts_pix[0, f, 2]
146
+ z0, z1, z2 = face_z[0, f, 0], face_z[0, f, 1], face_z[0, f, 2]
147
+ a0, a1, a2 = face_attrs[0, f, 0], face_attrs[0, f, 1], face_attrs[0, f, 2]
148
+
149
+ min_x = max(0, int(torch.floor(torch.min(torch.stack([v0[0], v1[0], v2[0]]))).item()))
150
+ max_x = min(width - 1, int(torch.ceil(torch.max(torch.stack([v0[0], v1[0], v2[0]]))).item()))
151
+ min_y = max(0, int(torch.floor(torch.min(torch.stack([v0[1], v1[1], v2[1]]))).item()))
152
+ max_y = min(height - 1, int(torch.ceil(torch.max(torch.stack([v0[1], v1[1], v2[1]]))).item()))
153
+
154
+ if min_x > max_x or min_y > max_y:
155
+ continue
156
+
157
+ px = torch.arange(min_x, max_x + 1, device=device).float() + 0.5
158
+ py = torch.arange(min_y, max_y + 1, device=device).float() + 0.5
159
+ px_grid, py_grid = torch.meshgrid(px, py, indexing='xy')
160
+ points = torch.stack([px_grid.flatten(), py_grid.flatten()], dim=-1)
161
+
162
+ def edge_fn(va, vb, p):
163
+ return (p[..., 0] - va[0]) * (vb[1] - va[1]) - (p[..., 1] - va[1]) * (vb[0] - va[0])
164
+
165
+ area = edge_fn(v0, v1, v2)
166
+ if abs(area.item()) < 1e-8:
167
+ continue
168
+
169
+ w0 = edge_fn(v1, v2, points) / area
170
+ w1 = edge_fn(v2, v0, points) / area
171
+ w2 = edge_fn(v0, v1, points) / area
172
+
173
+ inside = (w0 >= 0) & (w1 >= 0) & (w2 >= 0)
174
+ if not inside.any():
175
+ continue
176
+
177
+ idx = torch.where(inside)[0]
178
+ pts, iw0, iw1, iw2 = points[idx], w0[idx], w1[idx], w2[idx]
179
+
180
+ interp_z = iw0 * z0 + iw1 * z1 + iw2 * z2
181
+ interp_attr = iw0.unsqueeze(-1) * a0 + iw1.unsqueeze(-1) * a1 + iw2.unsqueeze(-1) * a2
182
+
183
+ pix_x, pix_y = pts[:, 0].long(), pts[:, 1].long()
184
+
185
+ for i in range(len(idx)):
186
+ x, y, z = pix_x[i].item(), pix_y[i].item(), interp_z[i].item()
187
+ if z < depth_buf[0, y, x].item():
188
+ depth_buf[0, y, x] = z
189
+ features[0, y, x] = interp_attr[i]
190
+ face_idx[0, y, x, 0] = f
191
+
192
+ return features, face_idx
193
+
194
+ def texture_sample(uv, texture, mode='bilinear'):
195
+ grid = uv.clone()
196
+ grid[..., 0] = grid[..., 0] * 2 - 1
197
+ grid[..., 1] = (1 - grid[..., 1]) * 2 - 1
198
+ sampled = F.grid_sample(texture, grid, mode=mode, padding_mode='border', align_corners=False)
199
+ return sampled.permute(0, 2, 3, 1).unsqueeze(1)
200
+
201
+ # =============================================================================
202
+ # TEXTURED MESH MODEL
203
+ # =============================================================================
204
+ @dataclass
205
+ class MeshConfig:
206
+ shape_path: str = 'shapes/bunny.obj'
207
+ shape_scale: float = 0.6
208
+ dy: float = 0.25
209
+ texture_resolution: int = 512
210
+
211
+ class TexturedMeshModel(nn.Module):
212
+ def __init__(self, config: MeshConfig, render_size=256, cache_path=None, device='cpu'):
213
+ super().__init__()
214
+ self.device = device
215
+ self.config = config
216
+ self.dy = config.dy
217
+ self.mesh_scale = config.shape_scale
218
+ self.texture_res = config.texture_resolution
219
+ self.cache_path = cache_path
220
+
221
+ self.proj = perspective_projection(np.pi / 3)
222
+ self.mesh = Mesh(config.shape_path, device).normalize_mesh(True, config.shape_scale, config.dy)
223
+
224
+ texture = torch.ones(1, 3, self.texture_res, self.texture_res, device=device)
225
+ self.texture_img = nn.Parameter(texture)
226
+
227
+ self.vt, self.ft = self._init_uv()
228
+ self.face_attrs = self.vt.unsqueeze(0)[:, self.ft.long()]
229
+
230
+ def _init_uv(self):
231
+ if self.cache_path:
232
+ vt_path = Path(self.cache_path) / 'vt.pth'
233
+ ft_path = Path(self.cache_path) / 'ft.pth'
234
+ if vt_path.exists() and ft_path.exists():
235
+ return torch.load(vt_path).to(self.device), torch.load(ft_path).to(self.device)
236
+
237
+ if self.mesh.vt is not None and self.mesh.vt.shape[0] > 0:
238
+ return self.mesh.vt.to(self.device), self.mesh.ft.to(self.device)
239
+
240
+ import xatlas
241
+ v_np = self.mesh.vertices.cpu().numpy()
242
+ f_np = self.mesh.faces.int().cpu().numpy()
243
+
244
+ atlas = xatlas.Atlas()
245
+ atlas.add_mesh(v_np, f_np)
246
+ opts = xatlas.ChartOptions()
247
+ opts.max_iterations = 4
248
+ atlas.generate(chart_options=opts)
249
+
250
+ _, ft_np, vt_np = atlas[0]
251
+ vt = torch.from_numpy(vt_np.astype(np.float32)).to(self.device)
252
+ ft = torch.from_numpy(ft_np.astype(np.int64)).to(self.device)
253
+
254
+ if self.cache_path:
255
+ os.makedirs(self.cache_path, exist_ok=True)
256
+ torch.save(vt.cpu(), Path(self.cache_path) / 'vt.pth')
257
+ torch.save(ft.cpu(), Path(self.cache_path) / 'ft.pth')
258
+
259
+ return vt, ft
260
+
261
+ def render(self, theta, phi, radius, dims=None):
262
+ dims = dims or (RENDER_SIZE, RENDER_SIZE)
263
+ cam = camera_from_angles(torch.tensor(theta), torch.tensor(phi), radius, self.dy)
264
+
265
+ verts_cam, verts_img, normals = prepare_vertices(
266
+ self.mesh.vertices, self.mesh.faces, self.proj, cam)
267
+
268
+ depth_attr = verts_cam[:, :, :, -1:]
269
+ depth, _ = rasterize(dims[1], dims[0], verts_cam[:, :, :, -1], verts_img, depth_attr)
270
+
271
+ mask_d = depth != 0
272
+ if mask_d.any():
273
+ d_min, d_max = depth[mask_d].min(), depth[mask_d].max()
274
+ if d_max > d_min:
275
+ depth[mask_d] = 0.5 + 0.5 * (depth[mask_d] - d_min) / (d_max - d_min)
276
+
277
+ uv_feats, face_idx = rasterize(dims[1], dims[0], verts_cam[:, :, :, -1], verts_img, self.face_attrs)
278
+ mask = (face_idx > -1).float()
279
+
280
+ img_feats = texture_sample(uv_feats, self.texture_img).squeeze(1) * mask
281
+ img_feats = img_feats + (1 - mask)
282
+
283
+ return {
284
+ 'image': img_feats.permute(0, 3, 1, 2).clamp(0, 1),
285
+ 'mask': mask.permute(0, 3, 1, 2),
286
+ 'depth': depth.permute(0, 3, 1, 2),
287
+ 'render_cache': {'uv_features': uv_feats, 'face_idx': face_idx}
288
+ }
289
+
290
+ def export_mesh(self, path, name=''):
291
+ os.makedirs(path, exist_ok=True)
292
+ v_np = self.mesh.vertices.cpu().numpy()
293
+ f_np = self.mesh.faces.int().cpu().numpy()
294
+ vt_np = self.vt.cpu().numpy()
295
+ ft_np = self.ft.cpu().numpy()
296
+
297
+ tex = self.texture_img.permute(0, 2, 3, 1).clamp(0, 1)[0].detach().cpu().numpy()
298
+ Image.fromarray((tex * 255).astype(np.uint8)).save(f'{path}/{name}albedo.png')
299
+
300
+ with open(f'{path}/{name}mesh.obj', 'w') as fp:
301
+ fp.write(f'mtllib {name}mesh.mtl\n')
302
+ for v in v_np:
303
+ fp.write(f'v {v[0]} {v[1]} {v[2]}\n')
304
+ for v in vt_np:
305
+ fp.write(f'vt {v[0]} {v[1]}\n')
306
+ fp.write('usemtl mat0\n')
307
+ for i in range(len(f_np)):
308
+ fp.write(f"f {f_np[i,0]+1}/{ft_np[i,0]+1} {f_np[i,1]+1}/{ft_np[i,1]+1} {f_np[i,2]+1}/{ft_np[i,2]+1}\n")
309
+
310
+ with open(f'{path}/{name}mesh.mtl', 'w') as fp:
311
+ fp.write('newmtl mat0\nKa 1 1 1\nKd 1 1 1\nKs 0 0 0\nillum 1\n')
312
+ fp.write(f'map_Kd {name}albedo.png\n')
313
+
314
+ # =============================================================================
315
+ # SD PIPELINE (PyTorch + INT8 Quantization)
316
+ # =============================================================================
317
+ # NOTE: ONNX doesn't support Depth2Img pipeline (5 channels vs 4)
318
+ # Using PyTorch with INT8 quantization instead
319
+ sd_pipe = None
320
+
321
+ def load_pipeline():
322
+ global sd_pipe
323
+ if sd_pipe is not None:
324
+ return sd_pipe
325
+
326
+ print("\n[INFO] Loading SD-2-Depth pipeline (PyTorch + INT8)...")
327
+ print("[INFO] Note: ONNX not supported for Depth2Img (5-channel UNet)")
328
+ from diffusers import StableDiffusionDepth2ImgPipeline
329
+
330
+ try:
331
+ print(f"[1/2] Downloading {SD_MODEL}...")
332
+ sd_pipe = StableDiffusionDepth2ImgPipeline.from_pretrained(
333
+ SD_MODEL,
334
+ torch_dtype=DTYPE,
335
+ )
336
+ print("[OK] Model downloaded")
337
+
338
+ # Quantize on CPU for faster inference
339
+ if DEVICE == "cpu":
340
+ try:
341
+ from optimum.quanto import quantize, freeze, qint8
342
+ print("[2/2] Applying INT8 quantization to UNet...")
343
+ quantize(sd_pipe.unet, weights=qint8)
344
+ freeze(sd_pipe.unet)
345
+ print("[OK] INT8 quantization applied (3-5x faster than FP32)")
346
+ except ImportError:
347
+ print("[WARN] optimum.quanto not available, using FP32 (slower)")
348
+
349
+ sd_pipe = sd_pipe.to(DEVICE)
350
+
351
+ # Disable autocast on CPU to avoid color issues
352
+ if DEVICE == "cpu":
353
+ sd_pipe.set_progress_bar_config(disable=False)
354
+ # Force FP32 for VAE to prevent color artifacts
355
+ sd_pipe.vae = sd_pipe.vae.float()
356
+
357
+ print("[OK] Pipeline ready!")
358
+ return sd_pipe
359
+
360
+ except Exception as e:
361
+ print(f"[ERROR] Pipeline loading failed: {e}")
362
+ if "401" in str(e) or "token" in str(e).lower():
363
+ print("[ERROR] Authentication required. Set HF_TOKEN environment variable")
364
+ raise
365
+
366
+ # =============================================================================
367
+ # MAIN PIPELINE
368
+ # =============================================================================
369
+ def dilate_texture(tex, mask, iterations=50):
370
+ """Dilate texture to fill gaps using scipy morphological operations (fast)."""
371
+ from scipy import ndimage
372
+
373
+ result = tex.clone().detach().numpy()
374
+ filled = mask.clone().detach().numpy().astype(bool)
375
+
376
+ # Use scipy binary_dilation for speed
377
+ for _ in range(iterations):
378
+ if filled.all():
379
+ break
380
+
381
+ # Find boundary pixels (unfilled with filled neighbors)
382
+ dilated_mask = ndimage.binary_dilation(filled)
383
+ boundary = dilated_mask & ~filled
384
+
385
+ if not boundary.any():
386
+ break
387
+
388
+ # For each boundary pixel, average from filled neighbors
389
+ for c in range(3):
390
+ # Compute neighbor average using convolution
391
+ kernel = np.array([[1,1,1],[1,0,1],[1,1,1]], dtype=np.float32)
392
+ neighbor_sum = ndimage.convolve(result[c] * filled, kernel, mode='constant')
393
+ neighbor_count = ndimage.convolve(filled.astype(np.float32), kernel, mode='constant')
394
+
395
+ # Update boundary pixels
396
+ valid = boundary & (neighbor_count > 0)
397
+ result[c][valid] = neighbor_sum[valid] / neighbor_count[valid]
398
+
399
+ filled = filled | boundary
400
+
401
+ return torch.from_numpy(result).float(), torch.from_numpy(filled)
402
+
403
+ def project_to_texture(tex, gen_img, uv, mask, blend=0.7, uv_mask=None):
404
+ """Project generated image to UV texture using scipy interpolation."""
405
+ from scipy.interpolate import griddata
406
+
407
+ _, _, TH, TW = tex.shape
408
+ new_tex = tex.clone()
409
+
410
+ # Flatten arrays
411
+ mask_f = mask[0, 0].reshape(-1).detach().numpy()
412
+ uv_f = uv[0].reshape(-1, 2).detach().numpy()
413
+ gen_np = gen_img[0].permute(1, 2, 0).reshape(-1, 3).detach().numpy()
414
+
415
+ # Get visible pixels
416
+ vis = mask_f > 0.5
417
+ if vis.sum() < 10:
418
+ return new_tex, uv_mask
419
+
420
+ # UV coords of visible pixels (source points)
421
+ src_uv = uv_f[vis] # N x 2
422
+ src_colors = gen_np[vis] # N x 3
423
+
424
+ # Target UV grid (destination)
425
+ tx = np.linspace(0, 1, TW)
426
+ ty = np.linspace(0, 1, TH)
427
+ grid_x, grid_y = np.meshgrid(tx, ty)
428
+
429
+ # Flip V coordinate
430
+ src_uv_flipped = src_uv.copy()
431
+ src_uv_flipped[:, 1] = 1 - src_uv_flipped[:, 1]
432
+
433
+ # Interpolate each channel
434
+ proj_tex = np.zeros((TH, TW, 3), dtype=np.float32)
435
+ for c in range(3):
436
+ proj_tex[:, :, c] = griddata(
437
+ src_uv_flipped, src_colors[:, c],
438
+ (grid_x, grid_y), method='linear', fill_value=np.nan
439
+ )
440
+
441
+ # Create mask of valid (non-NaN) pixels
442
+ proj_mask = ~np.isnan(proj_tex[:, :, 0])
443
+ proj_tex = np.nan_to_num(proj_tex, nan=0.5)
444
+
445
+ # Track cumulative UV coverage
446
+ if uv_mask is None:
447
+ uv_mask = torch.zeros(TH, TW, dtype=torch.bool)
448
+
449
+ proj_mask_t = torch.from_numpy(proj_mask)
450
+ proj_tex_t = torch.from_numpy(proj_tex).permute(2, 0, 1).float()
451
+
452
+ # Blend
453
+ new_pixels = proj_mask_t & ~uv_mask
454
+ existing_pixels = proj_mask_t & uv_mask
455
+
456
+ for c in range(3):
457
+ new_tex[0, c][new_pixels] = proj_tex_t[c][new_pixels]
458
+ new_tex[0, c][existing_pixels] = blend * proj_tex_t[c][existing_pixels] + (1 - blend) * new_tex[0, c][existing_pixels]
459
+
460
+ uv_mask = uv_mask | proj_mask_t
461
+ return new_tex, uv_mask
462
+
463
+
464
+ def finalize_texture(tex, uv_mask, iterations=100):
465
+ """Fill remaining gaps in texture using dilation."""
466
+ # Extract texture as numpy
467
+ tex_np = tex[0].clone()
468
+
469
+ # Dilate to fill gaps
470
+ dilated, filled = dilate_texture(tex_np, uv_mask, iterations=iterations)
471
+
472
+ # Put back
473
+ result = tex.clone()
474
+ result[0] = dilated
475
+ return result
476
+
477
+ def generate_texture(mesh_file, prompt, num_views, num_steps, seed, progress=gr.Progress()):
478
+ if mesh_file is None:
479
+ raise gr.Error("Please upload a mesh file!")
480
+ if not prompt.strip():
481
+ raise gr.Error("Please enter a text prompt!")
482
+
483
+ temp_dir = tempfile.mkdtemp()
484
+
485
+ try:
486
+ mesh_ext = os.path.splitext(mesh_file)[1].lower()
487
+ mesh_path = os.path.join(temp_dir, f"mesh{mesh_ext}")
488
+ shutil.copy(mesh_file, mesh_path)
489
+
490
+ progress(0.1, desc="Creating UV map...")
491
+ config = MeshConfig(shape_path=mesh_path, texture_resolution=TEXTURE_RESOLUTION)
492
+ model = TexturedMeshModel(config, RENDER_SIZE, Path(temp_dir) / 'cache', 'cpu')
493
+
494
+ progress(0.2, desc="Loading SD-2-Depth...")
495
+ pipe = load_pipeline()
496
+
497
+ viewpoints = [(0.5, 0.0), (0.5, np.pi/2), (0.5, np.pi), (0.5, -np.pi/2), (0.2, 0.0), (0.8, 0.0)][:num_views]
498
+
499
+ with torch.no_grad():
500
+ model.texture_img.fill_(0.5) # Start with neutral gray instead of white
501
+
502
+ previews = []
503
+ for i, (theta, phi) in enumerate(viewpoints):
504
+ progress(0.3 + 0.5 * i / len(viewpoints), desc=f"View {i+1}/{len(viewpoints)}...")
505
+
506
+ result = model.render(theta, phi, 2.0, (RENDER_SIZE, RENDER_SIZE))
507
+ depth = result['depth'][0, 0].cpu().numpy()
508
+ mask = result['mask'][0, 0].cpu().numpy()
509
+
510
+ if mask.sum() > 0:
511
+ d_vis = depth[mask > 0]
512
+ d_min, d_max = d_vis.min(), d_vis.max()
513
+ if d_max > d_min:
514
+ depth = (depth - d_min) / (d_max - d_min)
515
+ depth = depth * mask
516
+
517
+ depth_img = Image.fromarray((np.clip(depth, 0, 1) * 255).astype(np.uint8)).convert('RGB')
518
+
519
+ gen = torch.Generator(device=DEVICE).manual_seed(int(seed)) # Same seed for consistency
520
+ # SD-2-Depth: native depth conditioning (same as original TEXTure)
521
+ steps = int(num_steps) if num_steps else NUM_INFERENCE_STEPS
522
+ direction = ["front", "right side", "back", "left side"][i % 4]
523
+ textured = pipe(
524
+ prompt=f"{prompt}, {direction} view, consistent style",
525
+ image=depth_img,
526
+ strength=0.85, # Slightly less strength for more depth adherence
527
+ num_inference_steps=steps,
528
+ guidance_scale=7.5,
529
+ generator=gen
530
+ ).images[0]
531
+ previews.append(textured)
532
+
533
+ uv = result['render_cache']['uv_features']
534
+ gen_t = torch.tensor(np.array(textured)).float().permute(2, 0, 1).unsqueeze(0) / 255.0
535
+
536
+ with torch.no_grad():
537
+ # Track UV coverage across views
538
+ if i == 0:
539
+ uv_mask = None
540
+ model.texture_img.data, uv_mask = project_to_texture(
541
+ model.texture_img, gen_t, uv, result['mask'],
542
+ blend=0.5, uv_mask=uv_mask
543
+ )
544
+
545
+ progress(0.85, desc="Filling gaps...")
546
+
547
+ # Final dilation to fill any remaining gaps
548
+ with torch.no_grad():
549
+ model.texture_img.data = finalize_texture(model.texture_img, uv_mask, iterations=150)
550
+
551
+ progress(0.9, desc="Saving...")
552
+
553
+ tex_np = model.texture_img[0].permute(1, 2, 0).clamp(0, 1).detach().numpy()
554
+ tex_img = Image.fromarray((tex_np * 255).astype(np.uint8))
555
+ tex_img.save(f'{temp_dir}/uv_texture.png')
556
+
557
+ # Render 3D preview with texture
558
+ preview_result = model.render(0.4, 0.3, 2.5, (512, 512))
559
+ preview_np = preview_result['image'][0].permute(1, 2, 0).clamp(0, 1).detach().cpu().numpy()
560
+ preview_img = Image.fromarray((preview_np * 255).astype(np.uint8))
561
+ previews.insert(0, preview_img) # Add 3D preview as first image
562
+
563
+ model.export_mesh(f'{temp_dir}/mesh', '')
564
+
565
+ zip_path = f'{temp_dir}/textured_mesh.zip'
566
+ with zipfile.ZipFile(zip_path, 'w') as zf:
567
+ for f in ['mesh/albedo.png', 'mesh/mesh.obj', 'mesh/mesh.mtl', 'uv_texture.png']:
568
+ if os.path.exists(f'{temp_dir}/{f}'):
569
+ zf.write(f'{temp_dir}/{f}', os.path.basename(f))
570
+
571
+ progress(1.0, desc="Done!")
572
+ return tex_img, previews, zip_path
573
+
574
+ except Exception as e:
575
+ raise gr.Error(f"Error: {str(e)}")
576
+
577
+ # =============================================================================
578
+ # GRADIO UI
579
+ # =============================================================================
580
+ with gr.Blocks(title="TEXTure CPU Lite") as demo:
581
+ gr.Markdown("""# TEXTure CPU Lite
582
+ Generate UV texture maps for 3D meshes using text prompts.
583
+
584
+ ⚠️ **Quality Notice:** This is a simplified CPU-only demo. Results are significantly worse than the [original TEXTure paper](https://texturepaper.github.io/TEXTurePaper/).
585
+
586
+ **Why it looks bad:**
587
+ - No Kaolin GPU rasterizer → using slow software renderer with lower precision
588
+ - No proper view weighting → seams between views are visible
589
+ - No texture inpainting → blotchy patches instead of smooth transitions
590
+ - No refinement passes → single-pass projection loses detail
591
+ - INT8 quantization on CPU → color artifacts possible
592
+
593
+ **For production quality:** Use the [original TEXTure repo](https://github.com/TEXTurePaper/TEXTurePaper) with a GPU.
594
+ """)
595
+
596
+ with gr.Row():
597
+ with gr.Column():
598
+ mesh_in = gr.File(label="3D Mesh (.obj, .stl, .ply, .glb)", file_types=[".obj", ".stl", ".ply", ".glb", ".off"])
599
+ prompt_in = gr.Textbox(label="Texture Prompt", placeholder="ceramic with blue and white pattern", lines=2)
600
+ with gr.Row():
601
+ views_in = gr.Slider(2, 6, value=4, step=1, label="Views")
602
+ steps_in = gr.Slider(5, 25, value=20, step=1, label="Steps (5=fast, 20=quality)")
603
+ with gr.Row():
604
+ seed_in = gr.Number(value=42, label="Seed", precision=0)
605
+ btn = gr.Button("Generate", variant="primary")
606
+ gr.Markdown("**CPU Time:** ~1.5 min/view @ 10 steps, ~3 min/view @ 20 steps")
607
+
608
+ with gr.Column():
609
+ tex_out = gr.Image(label="UV Texture", type="pil")
610
+ gallery_out = gr.Gallery(label="3D Preview + Generated Views", columns=2, height=250)
611
+ zip_out = gr.File(label="Download (ZIP)")
612
+
613
+ btn.click(generate_texture, [mesh_in, prompt_in, views_in, steps_in, seed_in], [tex_out, gallery_out, zip_out])
614
+
615
+ gr.Markdown("**Credits:** [TEXTure Paper](https://texturepaper.github.io/TEXTurePaper/), [SD-2-Depth](https://huggingface.co/radames/stable-diffusion-2-depth-img2img), [xatlas](https://github.com/jpcy/xatlas)")
616
+
617
+ if __name__ == "__main__":
618
+ demo.queue(max_size=2).launch(ssr_mode=False)
requirements.txt ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # TEXTure CPU Lite - Text-Guided 3D Texturing
2
+ # With INT8 quantization for CPU deployment
3
+
4
+ # Core ML
5
+ torch>=2.1.0
6
+ torchvision>=0.16.0
7
+
8
+ # Diffusion + ONNX Runtime
9
+ diffusers>=0.25.0
10
+ transformers>=4.36.0
11
+ accelerate>=0.25.0
12
+ safetensors>=0.4.0
13
+ optimum[onnxruntime]>=1.17.0 # ONNX export + CPU runtime
14
+ optimum-quanto>=0.2.0 # Fallback INT8 quantization
15
+
16
+ # 3D Mesh handling
17
+ trimesh>=4.0.0
18
+ xatlas>=0.0.9 # UV unwrapping
19
+
20
+ # Image processing
21
+ Pillow>=10.0.0
22
+ numpy>=1.24.0
23
+ scipy>=1.11.0
24
+
25
+ # Gradio
26
+ gradio>=6.2.0
27
+
28
+ # Utils
29
+ tqdm>=4.66.0
30
+ loguru>=0.7.0
31
+ huggingface-hub>=0.20.0
32
+ psutil>=5.9.0
shapes/bunny.obj ADDED
The diff for this file is too large to render. See raw diff