I mentioned back in 2018 some issues regarding vertex normals being distorted when corrective shapes are enabled, and also the exporter's complete failure to properly calculate shape key normals from an object that is exported with custom normals enabled. I am just making this post to bring attention to this fact as it still has not been fixed in even the most recent release, and makes working with custom normals and corrective shapes a huge pain, basically requiring a total switch to 3DS Max or Maya for proper export.

1. A mesh with custom normals enabled and shape keys will not interpolate the vertex normals for it's shape keys correctly in Source unless a Data Transfer modifier using the "Topology" setting is enabled, transferring normals from a duplicate object in the scene. Failure to set this before export will result in shading errors from incorrect normals. This issue was not present in SMD/VTA exports prior to 2.7

2. Corrective shape keys that trigger from 3 or more shapes (A_B_C) distort the vertex normals, even if no actual vertices end up changing their position in 3D space. This issue also appears on ANY kind of corrective shape if the base object has custom normals set during export. I recorded a video demonstrating this problem during a discussion with Game Zombie, author of the 3DS Max Source Plugins, which is currently the only way as far as I'm aware to get a proper export for these cases.

https://www.youtube.com/watch?v=PLwBWWwurhI

I would really like to see these issues fixed, as someone who works excessively with custom normals and more advanced shape key structures, it has been a major hindrance on my workflow.

As always, thank you for the consideration. I can provide more information if necessary.

I've been noticing for a while now sometimes I still see broken normals on shapes, while most of these issues have been fixed over the years, one thing is still definately broken here, any corrective shape that triggers with 3 or more other shapes (A+B+C) will distort the vertex normals of the corrected area. It doesnt seem to matter how complex or simple the shape is, the exporter just doesn't seem to calculate it right with 3 or more. I can stack as many A+B correctives on top of each other as I want but a shape with more than 2 always seems to break.

I assume theres no need for a pic as by now you're probably quite aware of what this issue looks like when it happens

I know this is an old topic but I just wanted to say that its triangulation that is causing this, apply custom normals with a data transfer from a duplicate, underneath a triangulation modifier set to used Fixed method for quads

In case anyone else is wondering about this topic, I discovered the issue can be circumvented entirely by duplicating the mesh, and using data transfer to transfer the normals from the dupe specifically with the "Topology" setting. Simply export with the modifier enabled.

Ahh okay, these controller lines always confused the hell out of me but I see what you're saying now. Never even crossed my mind to set it up that way. I'll utilize that, thank you.

Oh I completely forgot you could do that, but the corrector flexes then would still have to be triggered manually right? I suppose that is a solution, although far less elegant than having it automatically blend them in when the parent shapes become active.

Wether or not the VTA export is calculating normals as intended or not I am unsure of, they do store the direction of the normal in addition to the world position translation I think, so Blender's solutions for the flexed vertex normals before exporting is probably causing the problem.

Would it be possible to have an option to make the DMX export behave the same as the VTA, broken or not?

I'm testing a model that has heavily modified vertex normals on the face in order to achieve anime-style shading, the problem Im having is when the base normals are modified this much, any shape keys on the model end up with distorted normals that pretty much ruin the shading for the flexes.

I recorded a short video just to demonstrate: https://puu.sh/AcDld/1713314db3.mp4
(normals being calculated after triangulization)

I'm able to get around this by exporting a VTA file instead and compiling the old way, but I absolutely need to have corrector shapes for the kind of flex setup Im doing, which VTA has no support for.

This seems to just be how Blender handles vertex normals on shape keys, I'm wondering if there is any way at all around this issue, a way to possibly export the DMX file without the normals on the shape keys imploding?

I see, thanks for the heads up.

I'm also wondering, is it possible to export more than a single animation per model? The way its set up in the Source Tools as well as the generated QC implies you can add more than one but I haven't been able to work out what to do for that (The Option_VertexCacheAnimationFile: Flexes already defined error appears if more than one VTA file is referenced under $model)

Ah, ok.

Are you sure about the VTA file working on DMX models that already have flex data?
Compiler seems to disagree: