Draft: hiresfix for img2img

[light-and-ray]

Add Hires. fix option into img2img tab #2213. Useful for generating images with high denoise straight. E.g. photo -> painting

Current implementation has issues:

• doesn't work for inpaint. Both only masked and full picture
• wrong total steps num in console progress bar
• not implemented new feature with upscale image from gallery (and it won't be implemented for inpaint, I think. Only with regeneration with previos options + upscaler for img2img override, how I did in my sd-webui-replacer extension)
• XYZ plot
• paste fields
• I don't know what is about API. Does txt2img api support hires. fix? I don't see it

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[light-and-ray]

I've moved fields and methods connected with hires fix from Txt2Img class into parent class to not dublicate

[light-and-ray]

Here is problem with inpaint. I can't understand what I need to do, to fit upscaled area into input image. Someone who know, maybe @AUTOMATIC1111, can you fix it?

dev-stable-diffusion-webui/modules/sd_samplers_cfg_denoiser.py", line 113, in apply_blend
        blended_latent = current_latent * self.nmask + self.init_latent * self.mask
                         ~~~~~~~~~~~~~~~^~~~~~~~~~~~
    RuntimeError: The size of tensor a (128) must match the size of tensor b (64) at non-singleton dimension 3

[light-and-ray]

Just copied all staff, and changed elem_id s and denoise_straight -> hr_denoise_straight

[light-and-ray]

It looks like I forgot to change class name

[w-e-w]

I think it makes more sense to just extend loopback script with image upscaling
https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/8a6a4ad894c2e28bcb6924490396997bfed7f07f/scripts/loopback.py
or create a new script similer to loopback that is more specific to the usecase of "img2img hries fix"

hires fix in txt2img already adds a bunch to complexity to the pip line
so I don't think implementing this directly into the img2img pipeline is a good idea

[light-and-ray]

@w-e-w I see, I will try it. I will make alternative pr

[light-and-ray]

I thought it will hard to implement inpainting hires fix, but it is vice versa easier because I just need to override 1 variable in first pass (upscaler for img2img), and it's very easy for scripts

[light-and-ray]

I thought it will hard to implement inpainting hires fix, but it is vice versa easier because I just need to override 1 variable in first pass (upscaler for img2img), and it's very easy for scripts

[w-e-w]

I don't know what is about API. Does txt2img api support hires. fix? I don't see it

yes, look again
enable_hr denoising_strength hr_scale ......

[light-and-ray]

@w-e-w I have quickly explored loopback code, and understood few things:

1. It's selectable script. I think it's possible to make always on script, but it's really unnecessary for hires fix. I have no one selectable script, for which hires fix would be useful
2. There are a lot of things I need to add inside loopback, and It will be look too complicated and too specific for "loops = 2". So it's better to make it like extension or built-in script. I think second is better, because we can support new button to hires picture from gallery. And also we can make old hires fix for txt2img deprecated (Should we?)
3. For txt2img hires fix also is not necessary to be inside general ui, it can be selectable script . And I agree it looks like a crutch inside txt2img pipeline

So I decided to make a new pr with selectable script for img2img[/txt2img], called like pass into hiresfix automatically, and support of new ✨ button. It would be better then just copy hires.fix crutch from txt2img into img2img

[w-e-w]

And also we can make old hires fix for txt2img deprecated (Should we?)

I have concerted proposing to deprecate hires fix but drop the idea long ago
because I feel that it will cause chaos as lots of extensions already have different behaviors that is built on top of the current hires fix pipeline

I suggest that you join discord server (if you haven't already) and passed your ideas with the other developers and especially AUTO befor making lots of changes otherwise you work may go to waste

discord invire link is on wiki/Contributing

[light-and-ray]

One more thing, ✨ button takes options from ui. It can takes from selectable script, but it less intuitive. Mb if I call script just "hires fix options" it will be more intuitive

I suggest that you join discord server

okay

[strawberrymelonpanda]

@light-and-ray Curious if anything came from the discord discussions. I've wanted a ✨ button / HRFix for Img2img for awhile.

[light-and-ray]

@light-and-ray Curious if anything came from the discord discussions. I've wanted a ✨ button / HRFix for Img2img for awhile.

Nope nothing. Implementation in this pr is bad, because it duplicates txt2img hires fix, which is bad. Implementing new hires fix is okay, but nobody is interested

I'm not interested too, because for inpaint here it is hard to make, and my extension Replacer has improved hiresfix which now supports custom masks and fixing only one image from gallery. Maybe later I will implement hires fix for non-inpainting img2img as an extension of built-in script. Extension is more likely, because hires fix with no inpaint support looks bad for built-in feature

[strawberrymelonpanda]

Fair enough! Thanks for the info.

This draft might be better than nothing for now. I'll try out Replacer as well, maybe its hr-fix could be used stand-alone with a non-matching replacement prompt or something.