Skip to content

feat(node): view a device's 3D model, and place it in AR on the google flavor - #7639

Draft
jamesarich wants to merge 11 commits into
mainfrom
feat/device-3d-model
Draft

jamesarich wants to merge 11 commits into
mainfrom
feat/device-3d-model

Conversation

@jamesarich

@jamesarich jamesarich commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

The flasher now serves a real-size GLB beside each device illustration that has one (meshtastic/web-flasher#461, #462, and #463 for the rebuilt models). This lets a user see the hardware they're looking at in 3D, and on the google flavor stand it on a real surface at real size.

🌟 New Features

  • Node detail's hero avatar shows a 3D badge when a model exists; tapping it opens the model full screen, orbitable, framed from its bounds, with the illustration as the fallback.
  • The orbit view is lit by a bundled CC0 studio light probe (Poly Haven, Studio Small 09, 1.6 MB) over the theme background.
  • google flavor: a "View in your space" action places the model through ARCore. Models are in meters, so nothing is scaled; a later tap moves it, a one-finger drag anywhere turns it, and the placement ring hides once it's placed. Chips pick which face rests on the surface (upright, face up or down, either side). Real objects in front of the model hide it where the phone supports ARCore depth.
  • A bottom-center shutter saves the AR view, camera and model without the controls, to Pictures/Meshtastic (Android 10 and later, where that needs no storage permission).
  • The viewer and the AR view show the model's own copyright notice, read verbatim from the GLB's asset.copyright, behind an info button as the maps carry their credit.
  • fdroid and desktop have no AR (ARCore is proprietary) and never show the action. ARCore availability is checked when a model opens, never at app start.

🛠️ Refactoring & Architecture

  • New DeviceModelRepository (core:repository / core:data) over DeviceModelRemoteDataSource (core:network). The flasher answers a missing model with 200 and its HTML shell, so only model/gltf-binary counts as a model.
  • LocalDeviceModelArViewer in core:ui is the host seam: the google flavor provides an ARCore viewer, fdroid provides null.
  • The AR view is built on ARSceneView rather than SceneView's PlacementScene, which as of 4.56.0 doesn't pass depth occlusion through; it reuses the library's public tap-to-place, reticle, coaching and shadow-receiver pieces.
  • Android renders with SceneView directly; desktop and iOS use the multiplatform sceneview-compose SceneViewer. SceneViewer's Android HDR path frees its placeholder environment while the scene still draws it, so Android preloads the probe instead.
  • The orbit scene stays composed under the AR view with only its surface dropped, so coming back keeps the engine, compiled materials, probe and camera; only the model instance is rebuilt.
  • The viewer stays open across a rotation. The AR view holds the orientation while it is up, because a recreate starts a new ARCore session without the planes or the placed model.

🧹 Chores

  • Adds io.github.sceneview 4.56.0 (sceneview-compose, sceneview, and arsceneview on google only). Excludes the mac-only Compose runtime the desktop renderer drags in, and Fuel's kotlin-android-extensions-runtime, which duplicates parcelize classes.
  • The manifest keeps GLES 3.0 optional and removes VIBRATE, both merged in by SceneView, so neither narrows where the app installs.

Testing Performed

  • GlbBoundsTest: GLB header and accessor-bounds parsing, and reading the embedded copyright notice.
  • DeviceModelRemoteDataSourceTest: URL swap, a served GLB, and the flasher's HTML shell for a missing model.
  • SceneViewManifestTest: GLES 3.0 not required and VIBRATE absent in the merged manifest.
  • RestPoseTest: every rest pose sits flat on the surface and centered on the anchor, at the case's height, depth or width, with face up turning the screen to the sky.
  • TracerouteHistoryUseCaseTest passes the new model repository.
  • Manual: orbit viewer on an emulator (fdroid and google) and on desktop (macOS); on a Pixel 9 Pro, AR tap-to-place, the drag, rest poses, the return from AR and rotation in both views. ARCore cannot start a session on an arm64 emulator.

@coderabbitai

coderabbitai Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 15, 2026
@github-actions

This comment has been minimized.

@jamesarich
jamesarich force-pushed the feat/device-3d-model branch from a9aff7d to 8a9f288 Compare October 15, 2026 20:33
@github-actions

This comment has been minimized.

@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
9755 1 9754 0
View the top 1 failed test(s) by shortest run time
org.meshtastic.core.service.MeshNotificationManagerImplConversationTest::classMethod
Stack Traces | 0.01s run time
java.lang.AssertionError: Failed to fetch maven artifact org.robolectric:android-all-instrumented:10-robolectric-5803371-i7
	at org.robolectric.internal.dependency.MavenArtifactFetcher.fetchArtifact(MavenArtifactFetcher.java:130)
	at org.robolectric.internal.dependency.MavenDependencyResolver.lambda$getLocalArtifactUrls$0(MavenDependencyResolver.java:87)
	at org.robolectric.internal.dependency.MavenDependencyResolver.whileLocked(MavenDependencyResolver.java:107)
	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrls(MavenDependencyResolver.java:82)
	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrl(MavenDependencyResolver.java:123)
	at org.robolectric.plugins.LegacyDependencyResolver.getLocalArtifactUrl(LegacyDependencyResolver.java:95)
	at org.robolectric.plugins.DefaultSdkProvider$DefaultSdk.getJarPath(DefaultSdkProvider.java:145)
	at org.robolectric.internal.AndroidSandbox$SdkSandboxClassLoader.<init>(AndroidSandbox.java:128)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at org.robolectric.util.inject.Injector.inject(Injector.java:258)
	at org.robolectric.util.inject.Injector.lambda$memoized$1(Injector.java:239)
	at org.robolectric.util.inject.Injector$MemoizingProvider.get(Injector.java:506)
	at org.robolectric.util.inject.Injector.getInstanceInternal(Injector.java:230)
	at org.robolectric.util.inject.Injector.resolveDependencies(Injector.java:305)
	at org.robolectric.util.inject.Injector.inject(Injector.java:256)
	at org.robolectric.util.inject.Injector.lambda$memoized$1(Injector.java:239)
	at org.robolectric.util.inject.Injector$MemoizingProvider.get(Injector.java:506)
	at org.robolectric.util.inject.Injector.getInstanceInternal(Injector.java:230)
	at org.robolectric.util.inject.Injector.getInstance(Injector.java:214)
	at org.robolectric.util.inject.Injector$ScopeBuilderProvider.create(Injector.java:575)
	at org.robolectric.util.inject.Injector$ScopeBuilderProvider.lambda$get$0(Injector.java:558)
	at jdk.proxy3/jdk.proxy3.$Proxy20.build(Unknown Source)
	at org.robolectric.internal.SandboxManager.getAndroidSandbox(SandboxManager.java:68)
	at org.robolectric.RobolectricTestRunner.getAndroidSandbox(RobolectricTestRunner.java:283)
	at org.robolectric.RobolectricTestRunner.getSandbox(RobolectricTestRunner.java:261)
	at org.robolectric.RobolectricTestRunner.getSandbox(RobolectricTestRunner.java:67)
	at org.robolectric.internal.SandboxTestRunner.sandboxGroupingClassBlock(SandboxTestRunner.java:206)
	at org.robolectric.internal.SandboxTestRunner.classBlock(SandboxTestRunner.java:194)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
	at org.robolectric.RobolectricTestRunner.run(RobolectricTestRunner.java:149)
	at androidx.test.ext.junit.runners.AndroidJUnit4.run(AndroidJUnit4.java:162)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestExecutor.runRequest(JUnitTestExecutor.java:174)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestExecutor.accept(JUnitTestExecutor.java:84)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestExecutor.accept(JUnitTestExecutor.java:47)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestDefinitionProcessor.processTestDefinition(AbstractJUnitTestDefinitionProcessor.java:65)
	at org.gradle.api.internal.tasks.testing.SuiteTestDefinitionProcessor.processTestDefinition(SuiteTestDefinitionProcessor.java:53)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at org.gradle.internal.dispatch.MethodInvocation.invokeOn(MethodInvocation.java:77)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:28)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:19)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:88)
	at jdk.proxy1/jdk.proxy1.$Proxy4.processTestDefinition(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:178)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:126)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:122)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:72)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Connection reset
	at com.google.common.util.concurrent.AbstractFuture$Failure.newExecutionException(AbstractFuture.java:168)
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:298)
	at com.google.common.util.concurrent.AbstractFutureState.blockingGet(AbstractFutureState.java:250)
	at com.google.common.util.concurrent.Platform.get(Platform.java:54)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:263)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:96)
	at org.robolectric.internal.dependency.MavenArtifactFetcher.fetchArtifact(MavenArtifactFetcher.java:122)
	... 56 more
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:316)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:354)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:798)
	at java.base/java.net.Socket$SocketInputStream.implRead(Socket.java:974)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:964)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:999)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:963)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:878)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:851)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:328)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:388)
	at java.base/sun.net.www.MeteredStream.read(MeteredStream.java:134)
	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3334)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3327)
	at com.google.common.io.ByteStreams.copy(ByteStreams.java:115)
	at org.robolectric.internal.dependency.MavenArtifactFetcher$FetchToFileTask.call(MavenArtifactFetcher.java:255)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleAsyncTask.runInterruptibly(TrustedListenableFutureTask.java:164)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleAsyncTask.runInterruptibly(TrustedListenableFutureTask.java:147)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@jamesarich
jamesarich force-pushed the feat/device-3d-model branch from 8a9f288 to 05411f4 Compare October 16, 2026 15:50
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@jamesarich jamesarich added the ui Something to do with the UI of the Android app label Oct 16, 2026
…e flavor

The flasher serves a real-size GLB beside each device illustration that has
one. Node detail's hero avatar shows a 3D badge when the model exists and opens
it full screen, orbitable and lit by a bundled CC0 studio light probe over the
theme background, with the illustration as the fallback. On the google flavor
an AR action places the model on a real surface at real size through ARCore;
the fdroid flavor and desktop have no AR and never show it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request ui Something to do with the UI of the Android app

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant