NodeIO problem of use

[Oleksandr-Fedorov]

Hi DonMcCurdy, I should start by saying that I am a student working on a small project.I thank you for your monumental work in many areas, from gltf-viewer to this library for transforming gltf/glb files.
I took as a basis for the loading logic of your loader on the blob link where we drag&drop the folder with files and display the 3d model.Now I have added the ability of this library to display textures metalRough() the project itself I write on Next.js. I will be glad for any answer,I need to clarify a few of them.
Firstly I will provide you with the code:

import { useEffect, useState, useMemo, useCallback, useRef } from 'react'
import { LoaderUtils, Cache } from 'three'
import {
	GLTFLoader,
	GLTFParser,
} from 'three/examples/jsm/loaders/GLTFLoader.js'
import { FBXLoader, TGALoader } from 'three/examples/jsm/Addons.js'
import { DRACOLoader } from 'three/examples/jsm/Addons.js'
import { KTX2Loader } from 'three/examples/jsm/Addons.js'
import { LoadingManager, REVISION } from 'three'
import { useLoader, useThree } from '@react-three/fiber'
import { useMyContext } from '../../MyContext'
import { WebIO } from '@gltf-transform/core'
import {
	ALL_EXTENSIONS,
	EXTMeshoptCompression,
	KHRONOS_EXTENSIONS,
} from '@gltf-transform/extensions'
import { metalRough } from '@gltf-transform/functions'
import {
	MeshoptDecoder,
	MeshoptEncoder,
	MeshoptSimplifier,
} from 'meshoptimizer'
import draco3d from 'draco3dgltf'
//Function of creating a modal window for model with KTX2 texture error(just UI with 2 buttons OK-true and Cancel-false)
import createDialog from './ModalDialog'

interface LoaderProps {
	url: string
	rootPath: string
	assetMap: Map<string, File>
	fileType: string
	rootFile: File | string
}

const useModelLoader = ({
	url,
	rootPath,
	assetMap,
	fileType,
	rootFile,
}: LoaderProps): any => {
	const { gl, scene } = useThree()
	// Function for traverse materials(like in viewer.js file (DonMcCurdy))
	const traverseMaterials = useCallback(
		(object: any, callback: (mat: any) => void) => {
			object.traverse((node: any) => {
				if (!node.geometry) return
				const materials = Array.isArray(node.material)
					? node.material
					: [node.material]
				materials.forEach(callback)
			})
		},
		[]
	)

	// Function for cleanup(like in viewer.js file (DonMcCurdy) but also animations cleanup)
	const cleanup = useCallback(
		(model: any) => {
			if (!model) return

			
			scene.remove(model)

			
			model.traverse((node: any) => {
				if (node.geometry) {
					node.geometry.dispose()
				}
			})

			
			if (model.animations) {
				model.animations.forEach((animation: any) => {
					if (animation.clip) {
						animation.clip.dispose()
					}
				})
			}

		
			traverseMaterials(model, (material: any) => {
				// Если необходимо, можно вызвать material.dispose() и отдельно очищать текстуры
				if (material.dispose) {
					material.dispose()
				}
				for (const key in material) {
					if (
						key !== 'envMap' &&
						material[key] &&
						material[key].isTexture &&
						material[key].dispose
					) {
						material[key].dispose()
					}
				}
			})
		},
		[scene, traverseMaterials]
	)

	//Just to retern to main page if user cancel loading of model with KTX2 texture
	const { setIsViewerVisible } = useMyContext() as any
	const [content, setContent] = useState<any>({
		scene: null,
		clips: null,
	})
	const [errorOccurred, setErrorOccurred] = useState(false)

	// Create LoadingManager and loaders 1 time
	const MANAGER = useMemo(() => new LoadingManager(), [])
	const THREE_PATH = `https://unpkg.com/three@0.${REVISION}.x`

	const DRACO_LOADER = useMemo(
		() =>
			new DRACOLoader(MANAGER).setDecoderPath(
				`${THREE_PATH}/examples/jsm/libs/draco/gltf/`
			),
		[MANAGER]
	)

	const KTX2_LOADER = useMemo(
		() =>
			new KTX2Loader(MANAGER).setTranscoderPath(
				`${THREE_PATH}/examples/jsm/libs/basis/`
			),
		[MANAGER]
	)

	// Function for load model with KTX2 texture
	const loadModel = useCallback(
		async (url: string, loader: any, blobURLs: string[]) => {
			try {
				// Создаем экземпляр WebIO с необходимыми расширениями
				const io = new WebIO({ credentials: 'include' })
					.registerExtensions(ALL_EXTENSIONS)
					.registerDependencies({
						'meshopt.decoder': MeshoptDecoder,
						'meshopt.encoder': MeshoptEncoder,
					})

				console.log('Initializing WebIO:', io)

				const gltf_trans = await io.read(url)

				// Show dialog and wait for user's decision
				const continueLoading = await createDialog()

				if (!continueLoading) {
					setIsViewerVisible(false)
					blobURLs.forEach((url: any) => URL.revokeObjectURL(url))
					return null // Return null to indicate that loading was canceled
				}

				// like in example=============================
				await gltf_trans.transform(metalRough())

				const glb: any = await io.writeBinary(gltf_trans)

				if (loader) {
					return new Promise((resolve, reject) => {
						loader.parse(
							glb.buffer,
							'',
							(gltf: any) => {
								resolve({
									scene: gltf.scene,
									clips: gltf.animations || [],
								})
							},
							(error: any) => {
								reject(error)
							}
						)
					})
				}
				//==============================================
			} catch (error) {
				console.log('Ошибка загрузки модели:', error)
				setErrorOccurred(true)
			}
		},
		[setIsViewerVisible]
	)

	// Function for load model with fallback loader(like in viewer.js file (DonMcCurdy))
	const loadWithFallbackLoader = useCallback(
		async (url: string, otherLoader: any, fileType: string) => {
			return new Promise((resolve, reject) => {
				otherLoader.load(
					url,
					(object: any) => {
						console.log('Loaded model with fallback loader:', object)
						let loadedScene = null
						let clips = null

						if (fileType === 'gltf/glb') {
							loadedScene = object.scene || object.scenes[0]
							clips = object.animations || []
						} else if (fileType === 'fbx') {
							loadedScene = object
							clips = object.animations || []
						}

						if (!loadedScene) {
							reject(new Error('No scene found in the model'))
							return
						}

						resolve({ scene: loadedScene, clips })
					},
					undefined,
					(error: any) => {
						console.error('Error loading model with fallback:', error)
						reject(error)
					}
				)
			})
		},
		[]
	)
	const isLoadingStarted = useRef(false)
	useEffect(() => {
		if (!url) return

		const blobURLs: string[] = []

		MANAGER.setURLModifier((someUrl: string) => {
			const baseURL = LoaderUtils.extractUrlBase(url)
			const normalizedURL =
				rootPath +
				decodeURI(someUrl)
					.replace(baseURL, '')
					.replace(/^(\.?\/)/, '')

			if (assetMap.has(normalizedURL)) {
				const blob = assetMap.get(normalizedURL)
				const blobURL = URL.createObjectURL(blob!)
				blobURLs.push(blobURL)
				return blobURL
			}
			return someUrl
		})

		let loader: GLTFLoader | FBXLoader | null = null
		let otherLoader: GLTFLoader | FBXLoader | null = null

		if (fileType === 'gltf/glb') {
			loader = new GLTFLoader(MANAGER).setMeshoptDecoder(MeshoptDecoder)
			otherLoader = new GLTFLoader(MANAGER)
				.setCrossOrigin('anonymous')
				.setDRACOLoader(DRACO_LOADER)
				.setKTX2Loader(KTX2_LOADER.detectSupport(gl))
				.setMeshoptDecoder(MeshoptDecoder)

			MANAGER.onLoad = () => {
				console.log('All textures loaded successfully')
			}

			MANAGER.onError = url => {
				console.error('Error loading texture:', url)
			}
		} else if (fileType === 'fbx') {
			otherLoader = new FBXLoader(MANAGER).setCrossOrigin('anonymous')
		}

		const loadModelAsync = async () => {
			try {
				//First try to load model with KTX2 textures
				const result = await loadModel(url, loader, blobURLs)
				if (result) {
					setContent(result)
				} else {
					try {
						const fallbackResult = await loadWithFallbackLoader(
							url,
							otherLoader,
							fileType
						)
						setContent(fallbackResult)
					} catch (fallbackError) {
						console.error('All loaders failed:', fallbackError)
					}
				}
			} catch (error) {
				console.error('Primary loader failed, trying fallback:', error)
			} finally {
				// Clean up resources
				blobURLs.forEach(url => URL.revokeObjectURL(url))
			}
		}

		loadModelAsync().finally(() => {
			isLoadingStarted.current = false
		})

		return () => {
			// Clean up resources(once again XD for sure)
			blobURLs.forEach(url => URL.revokeObjectURL(url))
			DRACO_LOADER.dispose()
			KTX2_LOADER.dispose()
			useLoader.clear(GLTFLoader, url)
			if (content.scene) cleanup(content.scene)
			Cache.clear()
		}
	}, [url])

	return content
}

export default useModelLoader

The question that bothers me the most consists of 2 sub-questions.I have read the documentation, as you can see in the example I am working with WebIO but I noticed that NodeIO has additional features that can be used.Bottom line, the problem is that :
1.1)I tried to “switch” from WebIO to NodeIO
I dont know is it right solution,but i made it like this
"route.ts"

import { NextResponse } from 'next/server'
import { NodeIO } from '@gltf-transform/core'
import { ALL_EXTENSIONS } from '@gltf-transform/extensions'
import { metalRough } from '@gltf-transform/functions'
import { MeshoptDecoder, MeshoptEncoder } from 'meshoptimizer'

export async function POST(req: Request) {
   try {
   	
   	const { url } = await req.json()

   	// Validate URL input
   	if (!url || typeof url !== 'string') {
   		return NextResponse.json(
   			{ error: 'Invalid or missing URL' },
   			{ status: 400 }
   		)
   	}

   	const io = new NodeIO({ credentials: 'include' })
   		.registerExtensions(ALL_EXTENSIONS)
   		.registerDependencies({
   			'meshopt.decoder': MeshoptDecoder,
   			'meshopt.encoder': MeshoptEncoder,
   		})

   	
   	const document = await io.read(url)
   	await document.transform(metalRough())

   	const modelBuffer = await io.writeBinary(document)


   	return NextResponse.json(
   		{ model: modelBuffer }, 
   		{ status: 200 }
   	)
   } catch (error) {
   	console.error('Error loading model:', error)

   	// Handle different types of errors
   	if (error instanceof TypeError) {
   		return NextResponse.json(
   			{ error: 'Invalid URL or network error' },
   			{ status: 400 }
   		)
   	}

   	// Generic server error
   	return NextResponse.json({ error: 'Failed to load model' }, { status: 500 })
   }
}

and in the Loader method i changed only KTX2_Loader_func

const loadModel = useCallback(
		async (url: string, loader: any, blobURLs: string[]) => {
			try {
				// Создаем экземпляр WebIO с необходимыми расширениями
				const response = await api.post('/api/getloader', { url })
				debugger
				const result = await new Promise((resolve, reject) => {
					loader.parse(
						response.data.model.buffer,
						'',
						(gltf: any) =>
							resolve({
								scene: gltf.scene,
								clips: gltf.animations || [],
							}),
						reject
					)
				})

				setContent(result)
				//==============================================
			} catch (error) {
				console.log('Ошибка загрузки модели:', error)
				setErrorOccurred(true)
			}
		},
		[setIsViewerVisible]
	)

but this option does not work, although there is a minimal change in the logic I know,that blob url is created on client-browser,so such url can't be accepted in Node, but how can I make it the other way?

I read that loaders on the server is very problematic to implement, is it related ?
1.2)Am I right to understand that when I already get the glb/glTF object through GLTF parser/loader I need to do something similar to how you optimize the model await document.transform(
// Remove duplicate meshes, materials, textures, etc.
dedup(), ...)
After loading it, I need to convert my glb/glTF back to Document, I suspect I will have to convert the file to Uint8Array first, but how is this possible.

One last question, when dragging and dropping model files, how do I realise that the model ‘Requires spec/gloss materials (KHR_materials_pbrSpecularGlossiness), which this viewer cannot display. Materials will be converted to metal/rough.’ I noticed that after const gltf_trans = await io.read(url) there is ‘failed to fetch ex.’. And that's the only reason I could switch to another normal loader as you can see in the first code.

That's all, I apologise if it seems silly or illogical what I'm doing, I'm just trying to somehow understand the logic of your applications and do something for myself.

[donmccurdy]

Hi @Oleksandr-Fedorov! I'm not sure if I understand the flow of data through your application, is this what you had in mind?

1. User drops .gltf/.glb model into web page
2. Webpage uploads model to Node.js server by POST request
3. Node.js server optimizes model, converts spec/gloss to metal/rough, sends model back to webpage as response to POST request
4. Webpage displays the now-optimized model

If so, here are a few comments:

• Most of glTF Transform's functionality works in Node.js or in a web browser. The main exception is texture compression: KTX2 conversion requires a Node.js environment, WebP/PNG/JPEG optimization works best in a Node.js environment. If you aren't optimizing textures then you could simplify your process and bandwidth costs by keeping the model in-memory on the webpage and not uploading it to the server.
• It's not possible to send a Blob URL between two different JavaScript environments, like the server and the webpage. Data URLs would work, but would add +33% to filesize, this isn't ideal for large files. Ideally you'd send the Uint8Array or ArrayBuffer directly – but note that these binary arrays are not JSON. I'm not very familiar with Next.js but you'd need to investigate how to send/receive binary or Blob objects in Next.js.
• On https://gltf.report/ the Materials will be converted to metal/rough behavior is implemented by (1) loading the files into a Document with WebIO, (2) checking the extensions, (3) converting with metalRough(), and (4) rendering in three.js with @gltf-transform/view. That last step is more complex than using GLTFLoader, but you could instead write to a binary Uint8Array/ArrayBuffer as step (4) and then load in GLTFLoader.parse as you're already doing. Then there's no need to send anything to a server.

[Oleksandr-Fedorov]

Thank you for your reply.
I will try to make my point more clearly.
Initially,the logic of the online 3D viewer was based on simplifying the logic of your example,i.e. so that the client opens a page,drags files there and the model is displayed.It works optimally with WebIO,i.e. for me the initial problem was the non-display of custom textures,this was solved with WebIO,moreover,even when loading the model I could simplify certain textures/unite meshes using .transform(...) But, I saw that NodeIO has more functionality, moreover, due to my inexperience, it seemed to me that if the processing of model loading/optimisation logic will be done on the server, it will be more correct and better, and besides, my teacher said that any web application should have a server, and even a 3D viewer, as an architectural minimum XD.

To sum it up, I need to clarify your important opinion on my possibly illogical questions regarding NodeIO, so that I can understand you correctly.

1. In order for me to use NodeIO, at a minimum, in the case of using drag&drop files on the client, I must somehow forward this data to the server based on the received files and blobs, and not in the same way as the loading and normalization of blobs was processed on the client in your viewer-project, that is, for example, by the POST method
   Something like this : sending to the server

async function handleFileUpload(event: React.ChangeEvent<HTMLInputElement>) {
if (!event.target.files || event.target.files.length === 0) {
return; // User canceled file selection
}

const formData = new FormData();

for (const file of Array.from(Dropzonefiles)) {
formData.append('files', file);
}

formData.append('otherData', 'some data');

await fetch('/api/file', {
method: 'POST',
body: formData,
});
}

If there is a need to save model files, its processing, then save in the DB/Cloud. Here I also argued with the teacher that hypothetically if I have many users and each saves files of 100 MB or more in the DB, then no DB will help me with HD.

And then from the DB/cloud they should be removed for processing on the server and then, as without the DB, from the Node server send to the client as a Post request.

there is also a small problem here, but I think it should be done something like this (I hope)

from serv ->client

document.transform(simplify({ simplifier: MeshoptSimplifier, ratio: 0.75, error: 0.001 }),...,)
const modelBuffer = await io.writeBinary(document)

		// Return the model buffer
		return NextResponse.json(
			{ model: somehow make it binary },//send on client edited model in binary/or some other correct format
			{ status: 200 }
		)

2. I noticed that after loading the model by pressing the button you have the ability to optimize it, that is, in this case, I will need to store the Document object somewhere that I initially loaded to the client/server, so that at any time I can do .transform? Or I can somehow from the result obtained after parsing:

const result = await new Promise((resolve, reject) => {
loader.parse(

Somehow get the Document type and accordingly its .transform

And the last but not least important question, when loading the model, you said (1) loading the files into a Document with WebIO/NodeIO, (2) checking the extensions,
Item 1 as I understood it is roughly
const doc = await io.read(modelURL)

But here is the second item, I am a little unclear about where it is possible to check the loaded model for the presence of custom extensions. Perhaps the second item is something like

const DENYLIST = 'KHRMaterialsPBRSpecularGlossiness';

for (const extension of document.getRoot().listExtensionsUsed()) {
if (extension.extensionName===DENYLIST) {
await showWarningPanel()
...
}
}

That's all, you know, I want to say that maybe I don't know everything, but even if I don't have a server, I won't be upset to show my project at the presentation of my thesis, thank you for this opportunity.

Sorry again if I'm taking up your time with my questions, thank you for your work.🤝

[donmccurdy]

... it will be more correct and better, and besides, my teacher said that any web application should have a server, and even a 3D viewer, as an architectural minimum XD.

That's fine then, let's assume the processing should be server-side for your application.

... I must somehow forward this data to the server based on the received files and blobs ...

I would suggest starting first with just support for self-contained files like .glb, and only later adding support for folders or .zip archives, just to get the basic pipeline working first. Or even simpler, perhaps start by just trying to send a simple typed array — not a JSON array — like new Uint8Array([1, 2, 3, 4]) from the webpage to the server and back. See: https://stackoverflow.com/a/48568899

... Here I also argued with the teacher that hypothetically if I have many users and each saves files of 100 MB or more in the DB, then no DB will help me with HD.

In a production application I would put large binary resources into object storage (such as Google Cloud Storage or Amazon S3) rather than a relational database, and then store references to those in my relational database like Postgres. But for small projects and learning or prototyping purposes, that's overkill.

I will need to store the Document object somewhere that I initially loaded to the client/server, so that at any time I can do .transform? Or I can somehow from the result obtained after parsing...

I'm not sure if I followed this question entirely, but the 'Document' is an in-memory representation of a glTF file used for processing/editing by glTF Transform. This representation can be converted to/from an actual glTF file using the I/O classes. When you parse/load the model in THREE.GLTFLoader, you're parsing the glTF file and creating from it a three.js scene graph. These three types (glTF Transform Document, glTF file, and three.js scene graph) are distinct, you can't apply glTF Transform optimizations directly to a three.js scene graph for example. Only the glTF file (not the in-memory Document or scene graph) can be transferred over a network.

I am a little unclear about where it is possible to check the loaded model for the presence of custom extensions...

Yes, your code example looks good! If you then run the metalRough() transform, the extension will be removed.