MOSTRAmi | Interactive Guide & Live Simulator

Onboarding & Hardware Diagnostics

At first launch, the MOSTRAmi application runs an edge-side hardware diagnostic to check if the device satisfies the minimum requirements for executing on-device local AI (Gemma 4 LiteRT) and running ARCore features. If device resources are limited, the app gracefully falls back to cloud APIs.

                        Kotlin (DeviceCheckScreen.kt)
                        fun checkDevice(context: Context): DeviceCheckResult {

                          val totalRamGB = getAvailableRAM(context)

                          val canRunE4B = totalRamGB >= 3.8 // Gemma 4 requires 4GB RAM

                          return DeviceCheckResult(

                            hasCamera = checkCameraPermission(context),

                            hasARCore = isARCoreInstalled(context),

                            canRunE4B = canRunE4B

                          )

                        }

✓

RAM Allocation Optimization Checks RAM to allocate either the local E4B (4.2GB) or E2B (2.1GB) model.

✓

ARCore Verification Verifies augmented reality framework compatibility before launching the camera view.

Simulator Controls

Simulate the initial hardware tests. Click to run the diagnostic checks.

Semantic Anchoring & OCR Pipeline

The key innovation of MOSTRAmi lies in its two-stage pipeline: the visitor first scans the explanatory label (plaque) of the artwork. On-device OCR instantly extracts author, title, and year. This data is injected into the AI context as a **"Semantic Anchor"**, reducing unsupported identifications common to generalist VLM models.

                        Kotlin (CameraViewModel.kt)
                        ocrProcessor.analyzeImage(bitmap) { ocrResult ->

                          _scanState.value = ScanState.OCRComplete(ocrResult)

                          // Semantic Anchor injected into VLM context:

                          router.setSemanticAnchor(ocrResult.artist, ocrResult.title)

                        }

✓

On-Device ML Kit OCR High-speed optical character recognition processed locally (supports CJK characters).

✓

Local DB Validation Matches extracted strings against the local DB to correct typos and avoid misinterpretations.

Simulator Controls

Align the label on the phone viewport at the left and press the shutter button to simulate OCR.

Profile Steering & AI Persona

MOSTRAmi dynamically tunes the narrative tone and interface layouts based on the active visitor profile. The **"Visitor Profile Wizard"** adapts VLM prompt constraints, text size (EAA compliance), and TTS vocal speed to match diverse visitor needs.

🧑

Adult (Critical Style) Focuses on neoplatonic symbols, historical contexts, color theory, and restorations.

👴

Senior (Accessible Style) Applies 1.3x scaled fonts, high contrasts (WCAG 2.1 AA), descriptive narratives, and slower audio.

🧒

Kids (Magic Style) Focuses on engaging fairytale narratives, metaphorical explanations, and treasure hunts.

Simulator Controls

Select Visitor Profile:

Global Clio Live, Local TTS & Whisper Mode

An audio guide reimagined: MOSTRAmi implements native Clio Live cloud audio with continuous listening, spoken responses and barge-in, now available as a floating assistant across the main app surfaces. Local voice options appear only in the local or expert context, while Live and TTS are mutually exclusive to avoid double audio. Local fallback uses **Piper Neural TTS** via `sherpa-onnx`; **"Whisper Mode"** tracks the proximity sensor and redirects narration to the earpiece.

                        Kotlin (TTSManager.kt)
                        override fun onSensorChanged(event: SensorEvent) {

                          val distance = event.values[0]

                          val isNear = distance < proximitySensor.maximumRange

                          if (isNear) {

                            switchToEarpieceAndLowerVolume() // Whisper Mode!

                            _state.value = TTSState.Speaking(whisperActive = true)

                          }

                        }

✓

Natural Vocal Profiles High-quality neural voice models in 5 languages (e.g., Ryan and Amy in EN, Riccardo and Paola in IT).

✓

Live Subtitles Real-time subtitles aligned with audio playback for visitors with hearing impairments (EAA/WCAG).

Simulator Controls

Proximity Sensor (Whisper Mode): Hold to ear

Museums, Guided Routes & Distance

The Museums section is no longer just a PDF list: it now offers loadable packs with audience, duration, artworks to find, and route steps. GPS remains optional and helps sort museums, show distance, and open directions, but the visit can also begin from a museum-curated pack.

                        Kotlin (MapScreen.kt)
                        val pack = museumExperiencePackages(museum, appLanguage).first()

                        exhibitionViewModel.startVisit(pack.title, pack.artworks)

                        navController.navigate(Screen.Camera.route)

                        // GPS remains optional for distance and directions

Simulator Controls

Distance from Florence (Uffizi): 250 km

BLE Indoor Location & Kalman Filter

Inside concrete museum walls where GPS fails, MOSTRAmi scans for Bluetooth Low Energy (iBeacon) signals. By passing RSSI readings through a **Kalman Filter** and performing multilateration, the app isolates the visitor's relative room position, automatically pulling descriptions of the nearest artworks.

                        Kotlin (IndoorLocationManager.kt)
                        val distance = calculateDistanceFromRssi(result.rssi, metadata.txPower)

                        val filteredDistance = kalmanFilter.update(distance)

                        if (filteredDistance < 2.0) {

                          triggerProximityEvent(metadata.artworkId) // Artwork close!

                        }

✓

Kalman Stabilization Dampens signal spikes caused by visitor movement or solid structures, securing position coordinates.

✓

Room Checklists Automatically lists highlighted masterpieces in the current room, tracking what you see.

✓

Clio TourSync™ & Telemetry Calculates group distances relative to the tour guide, emitting out-of-range alarms (>15m) on the guide's screen.

Simulator Controls

Position visitor in room:

Art Log, Timeline Scrapbook & Annotations

The **Post-Visit Explorer & Scrapbook** preserves scanned works in a personal travel log. It builds a vertical scrapbook timeline with personal voice notes (.aac), souvenir photos taken near the painting, trip statistics (scans, XP, stickers), and options to export Polaroid cards or summary PDFs. The same detail-exploration pattern now appears in chat: users can pinch, tap detected points of interest, or circle a detail crop and continue the conversation on that exact area.

                        Kotlin (ScholarAnnotation.kt)
                        data class ScholarAnnotation(

                          val id: Long = 0,

                          val scanId: Long,

                          val xCoord: Float, // relative coordinate 0f..1f

                          val yCoord: Float,

                          val noteText: String

                        )

Simulator Controls

Add Note in simulator:

*Click on the canvas crop at the left to select relative coordinates before placing the pin.*