ComfyUI 新版PuLID换脸

ComfyUI 新版PuLID换脸 ComfyUI 新版PuLID换脸 Modified January 29, 2025 解压后的模型应放置在 ComfyUI/models/insightface/models/antelopev2 中。 https://huggingface.co/MonsterMMORPG/tools/tree/main 如果你是用的旧版的flux的PuLID节点,可能会遇到一些问题，下面是解决的办法：错误解决期望所有张量都在同一个设备上错误原因： Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper CUDA native layer norm) 对应插件地址 https://github.com/sipie800/ComfyUI PuLID Flux Enhanced 将下面代码复制，粘贴替换原来的 Code block JSON then selects the last element, which is the largest face face info = sorted(face info, key=lambda x: (x.bbox[2] x.bbox[0]) (x.bbox[3] x.bbox[1]))[ 1] iface embeds = torch.from numpy(face info.embedding).unsqueeze(0).to(device, dtype=dtype) break else: No face detected, skip this image logging.warning(f'Warning: No face detected in image {str(i)}') continue get eva clip embeddings face helper.clean all() face helper.read image(image[i]) face helper.get face landmarks 5(only center face=True) face helper.align warp face() if len(face helper.cropped faces) == 0: No face detected, skip this image continue Get aligned face image align face = face helper.cropped faces[0] Convert bgr face image to tensor align face = image to tensor(align face).unsqueeze(0).permute(0, 3, 1, 2).to(device) parsing out = face helper.face parse(functional.normalize(align face, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]))[0] parsing out = parsing out.argmax(dim=1, keepdim=True) bg = sum(parsing out == i for i in bg label).bool() white image = torch.ones like(align face) Only keep the face features if use gray: align face = to gray(align face) else: align face = align face face features image = torch.where(bg, white image, align face) Transform img before sending to eva clip Apparently MPS only supports NEAREST interpolation? face features image = functional.resize(face features image, eva clip.image size, transforms.InterpolationMode.BICUBIC if 'cuda' in device.type else transforms.InterpolationMode.NEAREST).to(device, dtype=dtype) face features image = functional.normalize(face features image, eva clip.image mean, eva clip.image std) eva clip id cond vit, id vit hidden = eva clip(face features image, return all features=False, return hidden=True, shuffle=False) id cond vit = id cond vit.to(device, dtype=dtype) for idx in range(len(id vit hidden)): id vit hidden[idx] = id vit hidden[idx].to(device, dtype=dtype) id cond vit = torch.div(id cond vit, torch.norm(id cond vit, 2, 1, True)) Combine embeddings id cond = torch.cat([iface embeds, id cond vit], dim= 1) Pulid encoder cond.append(pulid flux.get embeds(id cond, id vit hidden)) if not cond: No faces detected, return the original model logging.warning("PuLID warning: No faces detected in any of the given images, returning unmodified model.") return (model,) fusion embeddings if fusion == "mean": cond = torch.cat(cond).to(device, dtype=dtype) N,32,2048 if cond.shape[0] 1: cond = torch.mean(cond, dim=0, keepdim=True) elif fusion == "concat": cond = torch.cat(cond, dim=1).to(device, dtype=dtype) elif fusion == "max": cond = torch.cat(cond).to(device, dtype=dtype) if cond.shape[0] 1: cond = torch.max(cond, dim=0, keepdim=True)[0] elif fusion == "norm id": cond = torch.cat(cond).to(device, dtype=dtype) if cond.shape[0] 1: norm=torch.norm(cond,dim=(1,2)) norm=norm/torch.sum(norm) cond=torch.einsum("wij,w ij",cond,norm).unsqueeze(0) elif fusion == "max token": cond = torch.cat(cond).to(device, dtype=dtype) if cond.shape[0] 1: norm=torch.norm(cond,dim=2) ,idx=torch.max(norm,dim=0) cond=torch.stack([cond[j,i] for i,j in enumerate(idx)]).unsqueeze(0) elif fusion == "auto weight": 🤔 cond = torch.cat(cond).to(device, dtype=dtype) if cond.shape[0] 1: norm=torch.norm(cond,dim=2) order=torch.argsort(norm,descending=False,dim=0) regular weight=torch.linspace(fusion weight min,fusion weight max,norm.shape[0]).to(device, dtype=dtype) cond=[] for i in range(cond.shape[1]): o=order[:,i] cond.append(torch.einsum('ij,i j',cond[:,i,:],regular weight[o])) cond=torch.stack( cond,dim=0).unsqueeze(0) elif fusion == "train weight": cond = torch.cat(cond).to(device, dtype=dtype) if cond.shape[0] 1: if train step 0: with torch.inference mode(False): cond = online train(cond, device=cond.device, step=train step) else: cond = torch.mean(cond, dim=0, keepdim=True) sigma start = model.get model object("model sampling").percent to sigma(start at) sigma end = model.get model object("model sampling").percent to sigma(end at) Patch the Flux model (original diffusion model) Nah, I don't care for the official ModelPatcher because it's undocumented! I want the end result now, and I don’t mind if I break other custom nodes in the process. 😄 flux model = model.model.diffusion model Let's see if we already patched the underlying flux model, if not apply patch if not hasattr(flux model, "pulid ca"): Add perceiver attention, variables and current node data (weight, embedding, sigma start, sigma end) The pulid data is stored in Dict by unique node index, so we can chain multiple ApplyPulidFlux nodes! flux model.pulid ca = pulid flux.pulid ca flux model.pulid double interval = pulid flux.double interval flux model.pulid single interval = pulid flux.single interval flux model.pulid data = {} Replace model forward orig with our own new method = forward orig. get (flux model, flux model. class ) setattr(flux model, 'forward orig', new method) Patch is already in place, add data (weight, embedding, sigma start, sigma end) under unique node index flux model.pulid data[unique id] = { 'weight': weight, 'embedding': cond, 'sigma start': sigma start, 'sigma end': sigma end, } Keep a reference for destructor (if node is deleted the data will be deleted as well) self.pulid data dict = {'data': flux model.pulid data, 'unique id': unique id} return (model,) def del (self): Destroy the data for this node if self.pulid data dict: del self.pulid data dict['data'][self.pulid data dict['unique id']] del self.pulid data dict NODE CLASS MAPPINGS = { "PulidFluxModelLoader": PulidFluxModelLoader, "PulidFluxInsightFaceLoader": PulidFluxInsightFaceLoader, "PulidFluxEvaClipLoader": PulidFluxEvaClipLoader, "ApplyPulidFlux": ApplyPulidFlux, } NODE DISPLAY NAME MAPPINGS = { "PulidFluxModelLoader": "Load PuLID Flux Model", "PulidFluxInsightFaceLoader": "Load InsightFace (PuLID Flux)", "PulidFluxEvaClipLoader": "Load Eva Clip (PuLID Flux)", "ApplyPulidFlux": "Apply PuLID Flux", } 资料链接百度网盘通过网盘分享的文件：ComfyUI PuLID Flux ll 链接: https://pan.baidu.com/s/1dMLAb4s0vroO9Zk6ecKb7Q?pwd=cycy 提取码: cycy 来自百度网盘超级会员v5的分享说明文档 https://www.xiaobot.net/post/b98d5569 9a75 4b9b aac2 42a3e285be94 由于AI技术更新迭代，请以文档更新为准更多内容收录在⬇️ https://xiaobot.net/p/GoToComfyUI 网盘 https://pan.quark.cn/s/129886bbcc61 相关链接 ComfyUI PuLID Flux ll https://github.com/lldacing/ComfyUI PuLID Flux ll?tab=readme ov file guozinan/PuLID模型 https://huggingface.co/guozinan/PuLID/tree/main 解压后的模型应放置在 ComfyUI/models/insightface/models/antelopev2 中。 https://huggingface.co/MonsterMMORPG/tools/tree/main 如果你是用的旧版的flux的PuLID节点,可能会遇到一些问题，下面是解决的办法：错误解决期望所有张量都在同一个设备上错误原因： Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper CUDA native layer norm) 对应插件地址 https://github.com/sipie800/ComfyUI PuLID Flux Enhanced 将下面代码复制，粘贴替换原来的资料链接百度网盘通过网盘分享的文件：ComfyUI PuLID Flux ll 链接: https://pan.baidu.com/s/1dMLAb4s0vroO9Zk6ecKb7Q?pwd=cycy 提取码: cycy 来自百度网盘超级会员v5的分享说明文档 https://www.xiaobot.net/post/b98d5569 9a75 4b9b aac2 42a3e285be94 由于AI技术更新迭代，请以文档更新为准更多内容收录在⬇️ https://xiaobot.net/p/GoToComfyUI 网盘 https://pan.quark.cn/s/129886bbcc61 相关链接 ComfyUI PuLID Flux ll https://github.com/lldacing/ComfyUI PuLID Flux ll?tab=readme ov file guozinan/PuLID模型 https://huggingface.co/guozinan/PuLID/tree/main 新版的PuLID解决了模型污染的问题，所以建议用新版的。使用新版的ComfyUI PuLID Flux ll节点需要禁用或者删除之前的PuLID节点，不然可能会有冲突问题。 ComfyUI PuLID Flux ll 模型放在ComfyUI\models\pulid文件夹里面，模型注意用新版的。如果你是使用PuLID，还需要安装下面的模型。 EVA CLIP 是 EVA02 CLIP L 14 336，应该会自动下载（将位于 huggingface 目录中）。如果由于某些原因自动下载失败（并且你遇到 face analysis.py, init assert 'detection' in self.models 异常），请手动下载此 EVA CLIP 模型，将文件放入你的 ComfyUI/models/clip 目录并重启 ComfyUI。下载地址：https://huggingface.co/QuanSun/EVA CLIP/blob/main/EVA02 CLIP L 336 psz14 s6B.pt?download=true facexlib 依赖需要安装，模型将在首次使用时下载。最后你需要带有AntelopeV2的 InsightFace，