How do batteries create noise?
Batteries, like anything else electrical/electronic, get noise. This is physics.
(1) Johnson noise,
(2) shot noise (thermal noise)
(3) voltage noise
(4) current noise.
To cut the very long story short, according to a 1995 IEEE paper on "measurement of voltage noise in chemical batteries",
Ni-Cd battery gets the lowest noise - -195dBV/Hz at 1Hz & -204dBV/Hz at 6.5KHz
, (measaured spectrum 1Hz - 75KHz),
followed by D alkaline, then AA alkaline, then AA lithium...
I don't know why NiMH battery was not included in the test
Otherwise, I expect its noise level would have beaten NiCd.
In the paper, it recommended NiCd batteries to replace "noisy capacitor/resistor chains" used in biasing networks.
Something else may surprise you too:- resistor noise vs battery noise.
In a paper: "Measurement of battery noise & resistor-current noise at subaudio frequencies":-
It was found carbon resistors are most noisy - 10,000,000 times of its thermal noise ast 3Hz !
Standard (Zinc-carbon) dry batteries & wire-wound resistors "exhibit no measureable excess noise".
Though the internal resistance of a battery is low low well less than 1R, its inductance & capacitance may affect the passage of audio signal. So when used as HV for a SE classs A tube amp, & filament heating, like yr linestage, I'd use bypass caps for the audio signals to pass thru.
PS: That's why I use battery as cathode bias to replace the "noisy capacitor/resistor chain".