Introduction

Natural language control of autonomous systems represents one of the most compelling applications of Large Language Models in real life applications of LLMs. However, existing approaches face fundamental deployment challenges: cloud dependency creates privacy vulnerabilities, ongoing API costs scale prohibitively with usage, internet connectivity requirements limit operational flexibility, and even "edge" solutions may require desktop hardware that constrains true field portability.

Arial-Piloting addresses these challenges through an agentic architecture designed for real-world deployment. By combining specialized dual-model architectures with comprehensive autonomous decision-making systems, we demonstrate that sophisticated LLM-powered robotics can operate entirely at the edge while delivering enhanced capabilities 

Designed specifically to be run smoothly on middle to high tier consumer laptops, Arial-Piloting provides an exciting combination of robust processing, autonomous corrections, and adaptive LLM piloting. Together the reimagined system is capable of both providing users with an intuitive framework for piloting drones and solving many of the previous issues inherent in previous LLM based autonomous systems that were cloud dependent or required powerful workstations.

Approach

Building off of the basic Typefly architecture and their creation of the minispec drone flight language, Arial-Piloting reinvented the architecture to run locally with 5 specialized models, new minispec commands, new vision detection techniques, and a truly autonomous edge based LLM piloting system. The most important changes is the integration of VLMs, automatic replanning and completion assessment, a two part minispec generation architecture with specialized reasoning and writing models, improved vision recognition capabilities, and a focus on lightweight LLM enable true edge server performance. 

By creating a more adaptive program and combining it with carefully calibrated models, Arial-Piloting was able to create a robust piloting system. Through artificial dataset generation, fine tuning, and prompt engineering, Arial-Piloting was able to recreate and improve upon the capabilities demonstrated by Typefly’s original cloud dependent architecture. Marking an achievement for small model edge deployment capabilities as well as demonstrating the value of finetuned edge models over large few-shot models such as GPT-4. 

Demonstrations

The core innovation of Arial-Piloting lies in its multi-model architecture that enables high level reasoning and writing to be implemented entirely on an edge server. Through systematic experimentation across 130+ finetuned model iterations, it was found that splitting the process into reasoning (Qwen 3-4B) from code generation (Qwen 3-1.7B) eliminates the tendency of unified models to over-fixate on their environment rather than their task, be overly sensitive to changes in user prompts, and allows for the systems to generalize for a wider variety of tasks. The complete system orchestrates five specialized models: a classifier model for abstract object categories, dual reasoning and writing models for flight planning, a VLM for environment probing, and a dedicated assessment model for replanning decisions. Each model underwent either domain-specific fine-tuning or substantial prompt engineering rather than relying on general-purpose capabilities. Careful optimization and quantization enables the entire system to run LLMs exclusively on GPU with as little as 8GB VRAM.

The autonomous decision framework represents a fundamental advancement over traditional drone control systems. Basic automatic recentering occurs after every flight execution to maintain target visibility and optimal positioning, while a continuous replanning system using a dedicated Qwen 3-1.7B model enables real-time task assessment and adaptive behavior. For complex scene understanding beyond basic object detection, the system employs VLM-powered environment probing that can reason about abstract concepts and spatial relationships.

Edge computing optimization eliminates cloud dependencies entirely while achieving competitive performance metrics. The complete local processing pipeline delivers drone flight commands in 10-15 seconds on a laptop with 8GB VRAM and generated commands can continue for minutes depending on the task. This represents a true autonomous drone piloting system capable of processing in real time, autonomous replanning, and complex tasks all run entirely on an edge server.

Memory efficiency is achieved through quantized models and the usage of the Typefly study’s MiniSpec drone control language which optimizes the number of output tokens necessary. Real-world testing in challenging conditions including wind, poor lighting, and cluttered environments validates the system's robustness as a basis for practical deployment solutions.