I'll explain some mathematical idea behind it in in very simple terms, and than you could try to apply your knowledge of raytracing, but it is not going to be simple. It also depends on how realistic you are going to be in your model. The most realistic model is almost impossible to implement, so you might need some cheating, very usual in virtual reality though. :-)
Strictly speaking, for a fractal, the angle and the point of intersection between the surface and the ray of light and the surface of the fractal set is
truly undetermined.
This thing simply does not exist, even theoretically. How it can be? This is because the surface itself is something esoteric,; in particular, the surface measure is truly infinite. You should not be surprised too much when it comes to fractals.
[EDIT #3]
(To be able to define a
local normal, you always need to have something called
differentiable function. Please see:
http://en.wikipedia.org/wiki/Differentiable_function[
^].
This is never the case with a fractal. In fact, you don't need to have a fractal to face a non-differentiable function. There is an
uncountable set of non-differentiable functions not related to fractals in any sense of this word. The fractals are very special functions.)
[END EDIT #3]
When it comes to reflection of light, you should remember that in real-life physics it's possible to produce a piece of real material, something you can hold in your hand, which surface is best described using fractal mathematics. There are no true fractals in real life, but there are not such material objects as confined with a smooth surfaces as well: every solid matter under normal condition is composed of atoms, but for some object, fractal surface is just the best description. The fractal scaling goes down to a very small size, albeit not the size of the atom. And, it's important to understand that such objects have wonderful physical properties, including some optical properties.
If this characteristic size gets less then the light wavelength (which is the common case — the wavelength is something relatively big, you can safely consider it to be something like a half of micrometer), from the standpoint of optical property, such objects can behave like true fractals. There is nothing amazing about its look: typically, they just look as very, very black bodies; some surfaces were claimed to the blackest recorded objects. Therefore, the most realistic model of reflection would be the one considering diffraction of light on the fractal surface. I'm afraid you would go into nearly top-notch mathematical hardness if you want to study it:
http://www.sciencedirect.com/science/article/pii/S003960289800274X[
^],
http://www.researchgate.net/publication/1867777_Elastic_Scattering_by_Deterministic_and_Random_Fractals_Self-Affinity_of_the_Diffraction_Spectrum[
^],
http://www.researchgate.net/publication/225976935_Fractal_Characteristics_Investigation_on_Light_Scattering_from_Two_Dimensional_Rough_Surface[
^],
http://www.researchgate.net/publication/26315277_Diffraction_by_fractal_metallic_supergratings[
^].
Can it be simpler? Probably, but you will face just with unusual high volume of data processing (again, typical for fractal). The methods of raytracing use some approach not adequately describing real optics even for smooth surfaces not getting close to the wavelength. In the nature, there are no "rays", but you can model things using the idealized "rays" representing some narrow bunches of light. What is the nature of fractal surface? No one bunch is narrow enough. The whole idea of fractal is that it is scaled down to more and more details observed at higher magnification. If you can observe some total number of tiny feature at the limits of the resolution (say, visible as few pixels), and than increase the magnification, you will recognize as many feature on the magnified picture.
So, the cheating would be to re-render the fractal surface for each view of it, which is always done when fractals are visualized. This procedure always stops at some level of detail, otherwise you would calculate them infinitely. Then, you should evaluate the smallest characteristic feature of this "under-fractalled" surface. And use the "rays" of even smaller diameter. In other words, the diameter of a ray should shrink with the fractal view. In this case, you can use your usual methods of tracing. Not quite realistic, but probably would allow to get a better idea what a fractal is visually. I warned you, this is not easy at all.
[EDIT #1]
So the answers to your questions:
1 and 2: such things do not exist for fractal surface. They only exist for an "incomplete fractal", obtained from finite number of iteration used to build a model of a fractal. You can render such non-fractal surface for ever separate view and operate on this surface in a usual way.
3. Be happy if you can solve this problem even slowly. More seriously: the problem will incur unusually high volume of calculation. I would probably think about going in for using video card GPU.
[END EDIT #1]
[EDIT #2]
Perhaps my arbitrary use of the term "render" in "render of a surface" was confusing. This is absolutely not the same as your scene rendering. Here is what happens: with "normal surfaces" you can have a ready-to-track surface model and work with it when you want to render a scene for certain lighting and certain distance and orientation between a camera and a rendered object.
In fractal case, there is no such thing. A complete fractal object never exist in the computer, which is a finite-state machine. This is because representation of such objects "needs" infinite number of iteration, and its informational capacity is infinite. In a way, it only exists in our imagination. So, the rendering is two-stage: when you get a new distance and orientation between a camera and a rendered object, each time, you first need to build up an approximated model of the fractal surface, using some finite number of iteration. You should apply some criterion related to the level of detail you obtain by scaling into the fragment of this
self-similar object. On second state, you can ray-trace it.
[END EDIT #2]
So, wish you the best of luck,
—SA