Not found.

Paper on Continuous Collision Avoidance accepted to IROS 2019

Wolfgang Merkt, Vladimir Ivan, and Sethu Vijayakumar. “Continuous-Time Collision Avoidance for Trajectory Optimization in Dynamic Environments”, accepted to IEEE IROS 2019.

Publisher’s link – DOI: 10.1109/IROS40897.2019.8967641


Common formulations to consider collision avoidance in trajectory optimization often use either preprocessed environments or only check and penalize collisions at discrete time steps.
However, when only checking at discrete states, this requires either large margins that prevent manipulation close to obstacles or dense time discretization increasing the dimensionality of the optimization problem in complex environments. Nonetheless, collisions may still occur in the interpolation/transition between two valid states or in environments with thin obstacles.
In this work, we introduce a computationally inexpensive continuous-time collision avoidance term in presence of static and moving obstacles. Our penalty is based on conservative advancement and harmonic potential fields and can be used as either a cost or constraint in off-the-shelf nonlinear programming solvers.
Due to the use of conservative advancement (collision checks) rather than distance computations, our method outperforms discrete collision avoidance based on signed distance constraints resulting in smooth motions with continuous-time safety while planning in discrete time.
We evaluate our proposed continuous collision avoidance on scenarios including manipulation of moving targets, locomanipulation on mobile robots, manipulation trajectories for humanoids, and quadrotor path planning and compare penalty terms based on harmonic potential fields with ones derived from contact normals.


author={Merkt, Wolfgang and Ivan, Vladimir and Vijayakumar, Sethu},
booktitle={2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={Continuous-Time Collision Avoidance for Trajectory Optimization in Dynamic Environments},keywords={collision avoidance;discrete time systems;mobile robots;motion control;nonlinear programming;trajectory control;continuous-time safety;continuous collision avoidance;harmonic potential fields;trajectory optimization;dynamic environments;discrete time steps;conservative advancement;off-the-shelf nonlinear programming solvers;collision checks},

Paper on Shared Autonomy Applications using Whole-body Control Formulations of Locomanipulation accepted to CASE 2019

Wolfgang Merkt, Vladimir Ivan, Yiming Yang, and Sethu Vijayakumar. “Towards Shared Autonomy Applications using Whole-body Control Formulations of Locomanipulation”, accepted to IEEE CASE 2019.

Publisher’s link – DOI: 10.1109/COASE.2019.8843153


While widely studied in robotics for decades, mobile manipulation has recently seen a surge in interest for industrial applications due to increasing demands on flexibility and agility alongside productivity, particularly in small and medium enterprises. However, most mobile manipulation solutions frequently decouple the navigation from the manipulation problem effectively performing fixed-base manipulation using a repositionable manipulator. This is not only inefficient, but moreover limits the range of applications and disregards the inherent redundancy of a mobile manipulation system.

In this work, we introduce a high-performance omnidirectional mobile manipulation platform with integrated whole-body control, real-time collision-free whole-body motion planning, and perception. We demonstrate its capability along with application scenarios on technical demonstrators involving moving manipulation targets as well as whole-body manipulation in simulation and hardware experiments. Finally, we deploy and evaluate our solution in field trials on an industrial oil and gas training facility on a sensor placement and manipulation task.

A selection of videos related to this work can be found here.


author={Merkt, Wolfgang and Ivan, Vladimir and Yang, Yiming and Vijayakumar, Sethu},
title={\href{https://doi.org/10.1109/COASE.2019.8843153}{Towards Shared Autonomy Applications using Whole-body Control Formulations of Locomanipulation}},
keywords={collision avoidance;manipulators;mobile robots;path planning;sensor placement;whole-body control formulations;locomanipulation;industrial applications;flexibility;mobile manipulation solutions;fixed-base manipulation;repositionable manipulator;inherent redundancy;mobile manipulation system;high-performance omnidirectional mobile manipulation platform;real-time collision-free whole-body motion planning;moving manipulation;whole-body manipulation;industrial oil;gas training facility;sensor placement;manipulation task;shared autonomy applications;Planning;Task analysis;Robot kinematics;Optimization;Aerospace electronics;Manipulators},

Paper on Teleoperation Modes for Constrained Tasks accepted to CASE 2019

Christopher E. Mower, Wolfgang Merkt, Aled Davies, and Sethu Vijayakumar. “Comparing Alternate Modes of Teleoperation for Constrained Tasks”, accepted to IEEE CASE 2019.

Publisher’s link – DOI: 10.1109/COASE.2019.8843265


Teleoperation of heavy machinery in industry often requires operators to be in close proximity to the plant and issue commands on a per-actuator level using multiple joystick input devices. However, this is non-intuitive and makes achieving desired job properties a challenging task requiring operators to complete costly and extensive training.
Despite this, operator fatigue is common with implications for personal safety, project timeliness, cost, and job quality. While full automation is not yet achievable due to unpredictability and the dynamic nature of the environment and task, shared control paradigms allow operators to issue high-level commands in an intuitive, task-informed control space while having the robot optimize for achieving desired job properties.

In this paper, we compare a number of modes of teleoperation, exploring both the number of dimensions of the control input as well as the most intuitive control spaces. Our experimental evaluations of the performance metrics were based on quantifying the difficulty of tasks based on the well known Fitts’ law as well as a measure of how well constraints affecting the task performance were met. Our experiments show that higher performance is achieved when humans submit commands in low-dimensional task spaces as opposed to joint space manipulations.


@inproceedings{mower2019comparing, title={Comparing Alternate Modes of Teleoperation for Constrained Tasks}, booktitle={2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)}, author={Mower, Christopher Edwin and Merkt, Wolfgang and Vijayakumar, Sethu}, year={2019}, volume={}, number={}, pages={1497-1504}, keywords={industrial manipulators;machinery;telerobotics;teleoperation;intuitive control spaces;task performance;low-dimensional task spaces;constrained tasks;heavy machinery;per-actuator level;multiple joystick input devices;operator fatigue;personal safety;project timeliness;job quality;high-level commands;intuitive task-informed control space;job properties;Fitts’ law;Task analysis;Aerospace electronics;Spraying;Manipulators;Games;Measurement}, doi={10.1109/COASE.2019.8843265}, ISSN={2161-8089}, month={Aug} }

Paper on End-Pose Planning on Complex Support Surfaces accepted to Humanoids 2018

Henrique Ferrolho, Wolfgang Merkt, Yiming Yang, Vladimir Ivan, and Sethu Vijayakumar. “Whole-Body End-Pose Planning for Legged Robots on Inclined Support Surfaces in Complex Environments”, Proc. IEEE Intl. Conf. on Humanoid Robots (Humanoids 2018), Beijing (2018).

Publisher’s link – DOI: 10.1109/HUMANOIDS.2018.8625026


Planning balanced whole-body reaching configurations is a fundamental problem in humanoid robotics on which manipulation and locomotion planners depend on. While finding valid whole-body configurations in free space and on flat terrains is relatively straightforward, the problem becomes extremely challenging when obstacle avoidance is taken into account, and when balancing on more complex terrains, such as inclined supports or steps. Previous work using Paired Forward-Inverse Dynamic Reachability Maps demonstrated fast end-pose planning on flat terrains at different heights by decomposing the kinematic structure and leveraging combinatorics. In this paper, we present an efficient whole-body end-pose planning framework capable of finding collision-free whole-body configurations in complex environments and on sloped support regions. The main contributions in this paper are twofold: (i) the integration of contact property information of support regions into both precomputation and online planning stages, including whole-body static equilibrium robustness, and (ii) the proposal of a more informed and meaningful sampling strategy for the lower-body. We focus on humanoid robots throughout the paper, but all the principles can be applied to legged platforms other than bipedal robots. We demonstrate our method on the NASA Valkyrie humanoid platform with 38 Degrees of Freedom (DoF) over inclined supports. Analysis of the results indicate both higher success rates – greater than 95 % and 80 % on obstacle-free and highly cluttered environments, respectively – and shorter computation times compared to previous methods.


author={H. {Ferrolho} and W. {Merkt} and Y. {Yang} and V. {Ivan} and S. {Vijayakumar}},
booktitle={2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)},
title={Whole-Body End-Pose Planning for Legged Robots on Inclined Support Surfaces in Complex Environments},
keywords={Planning;Humanoid robots;Kinematics;Robustness;Task analysis;Collision avoidance},

Paper on Time-Configuration Space RRTConnect with Temporal Constraints accepted to Humanoids 2018

Yiming Yang, Wolfgang Merkt, Vladimir Ivan, and Sethu Vijayakumar. “Planning in Time-Configuration Space for Efficient Pick-and-Place in Non-Static Environments with Temporal Constraints”, Proc. IEEE Intl. Conf. on Humanoid Robots (Humanoids 2018), Beijing (2018).

Publisher’s link – DOI: 10.1109/HUMANOIDS.2018.8624989


This paper presents a novel sampling-based motion planning method using bidirectional search with a time-configuration space representation that is able to efficiently generate collision-free trajectories in complex and non-static environments. Our approach exploits time indexing to separate a complex problem with mixed constraints into multiple subproblems with simpler constraints that can be solved efficiently. We further introduce a planning framework by incorporating the proposed planning method enabling efficient pick-and-place of large objects in various scenarios. Simulation as well as hardware experiments show that the method also scales from redundant robot arms to mobile manipulators and humanoids. In particular, we have demonstrated that the proposed method is able to plan collision-free motion for a humanoid robot to pick up a large object placed inside a moving storage box while walking.


author={Y. {Yang} and W. {Merkt} and V. {Ivan} and S. {Vijayakumar}},
booktitle={2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)},
title={Planning in Time-Configuration Space for Efficient Pick-and-Place in Non-Static Environments with Temporal Constraints},
keywords={Planning;Grasping;Trajectory;Task analysis;Humanoid robots;End effectors},

Paper on Deep RL for Valkyrie accepted to Humanoids 2018

Chuanyu Yang, Kai Yuan, Wolfgang Merkt, Taku Komura, Sethu Vijayakumar, Zhibin Li. “Learning Whole-Body Motor Skills for Humanoids”, Proc. IEEE Intl. Conf. on Humanoid Robots (Humanoids 2018), Beijing (2018).

Publisher’s link – DOI: 10.1109/HUMANOIDS.2018.8625045


This paper presents a hierarchical framework for Deep Reinforcement Learning that acquires motor skills for a variety of push recovery and balancing behaviors, i.e., ankle, hip, foot tilting, and stepping strategies. The policy is trained in a physics simulator with realistic setting of robot model and low-level impedance control that are easy to transfer the learned skills to real robots. The advantage over traditional methods is the integration of high-level planner and feedback control all in one single coherent policy network, which is generic for learning versatile balancing and recovery motions against unknown perturbations at arbitrary locations (e.g., legs, torso). Furthermore, the proposed framework allows the policy to be learned quickly by many state-of-the-art learning algorithms. By comparing our learned results to studies of preprogrammed, special-purpose controllers in the literature, self-learned skills are comparable in terms of disturbance rejection but with additional advantages of producing a wide range of adaptive, versatile and robust behaviors.


author={C. {Yang} and K. {Yuan} and W. {Merkt} and T. {Komura} and S. {Vijayakumar} and Z. {Li}},
booktitle={2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)},
title={Learning Whole-Body Motor Skills for Humanoids},
keywords={Humanoid robots;Aerospace electronics;Collision avoidance;PD control;Reinforcement learning},

Paper on Warmstart Initialization for Trajectory Optimization accepted to IROS 2018

Wolfgang Merkt, Vladimir Ivan, and Sethu Vijayakumar. “Leveraging Precomputation with Problem Encoding for Warm-Starting Trajectory Optimization in Complex Environments”, Proc. IEEE Intl. Conf. on Intelligent Robotics (IROS 2018), Madrid (2018).

Publisher’s link – DOI: 10.1109/IROS.2018.8593977


Motion planning through optimization is largely based on locally improving the cost of a trajectory until an optimal solution is found. Choosing the initial trajectory has therefore a significant effect on the performance of the motion planner, especially when the cost landscape contains local minima. While multiple heuristics and approximations may be used to efficiently compute an initialization online, they are based on generic assumptions that do not always match the task at hand. In this paper, we exploit the fact that repeated tasks are similar according to some metric. We store solutions of the problem as a library of initial seed trajectories offline and employ a problem encoding to retrieve near-optimal warm-start initializations on-the-fly.
We compare how different initialization strategies affect the global convergence and runtime of quasi-Newton and probabilistic inference solvers. Our analysis on the 38-DoF NASA Valkyrie robot shows that efficient and optimal planning in high-dimensional state spaces is possible despite the presence of globally non-smooth and discontinuous constraints, such as the ones imposed by collisions.


author={W. {Merkt} and V. {Ivan} and S. {Vijayakumar}},
booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={Leveraging Precomputation with Problem Encoding for Warm-Starting Trajectory Optimization in Complex Environments},
keywords={collision avoidance;convergence;humanoid robots;mobile robots;Newton method;trajectory control;problem encoding;warm-starting trajectory optimization;motion planner;local minima;motion planning;near-optimal warm-start initializations;global convergence;quasiNewton solvers;probabilistic inference solvers;NASA Valkyrie robot;Task analysis;Collision avoidance;Planning;Robots;Trajectory optimization},

EXOTica accepted as a book chapter in the Springer ROS Book Vol. 3

Ivan V., Yang Y., Merkt W., Camilleri M.P., Vijayakumar S. (2019) EXOTica: An Extensible Optimization Toolset for Prototyping and Benchmarking Motion Planning and Control. In: Koubaa A. (eds) Robot Operating System (ROS). Studies in Computational Intelligence, vol 778. Springer, Cham


In this research chapter, we will present a software toolbox called EXOTica that is aimed at rapidly prototyping and benchmarking algorithms for motion synthesis. We will first introduce the framework and describe the components that make it possible to easily define motion planning problems and implement algorithms that solve them. We will walk you through the existing problem definitions and solvers that we used in our research, and provide you with a starting point for developing your own motion planning solutions. The modular architecture of EXOTica makes it easy to extend and apply to unique problems in research and in industry. Furthermore, it allows us to run extensive benchmarks and create comparisons to support case studies and to generate results for scientific publications. We demonstrate the research done using EXOTica on benchmarking sampling-based motion planning algorithms, using alternate state representations, and integration of EXOTica into a shared autonomy system. EXOTica is an opensource project implemented within ROS and it is continuously integrated and tested with ROS Indigo and Kinetic. The source code is available at https://github.com/ipab-slmc/exotica and the documentation including tutorials, download and installation instructions are available at https://ipab-slmc.github.io/exotica.


author=”Ivan, Vladimir and Yang, Yiming and Merkt, Wolfgang and Camilleri, Michael P. and Vijayakumar, Sethu”,
editor=”Koubaa, Anis”,
title=”EXOTica: An Extensible Optimization Toolset for Prototyping and Benchmarking Motion Planning and Control”,
bookTitle=”Robot Operating System (ROS): The Complete Reference (Volume 3)”,
publisher=”Springer International Publishing”,



HDRM: A Resolution Complete Dynamic Roadmap for Real-Time Motion Planning in Complex Scenes accepted to IEEE RA-L

Yiming Yang, Wolfgang Merkt, Vladimir Ivan, Zhibin Li, and Sethu Vijayakumar. “HDRM: A Resolution Complete Dynamic Roadmap for Real-Time Motion Planning in Complex Scenes”. IEEE Robotics and Automation Letters, 2018, In Press.

Publisher’s link – DOI: 10.1109/LRA.2017.2773669


We present the Hierarchical Dynamic Roadmap (HDRM), a novel resolution complete motion planning algorithm for solving complex planning problems. A unique hierarchical structure is proposed for efficiently encoding the configuration-to-workspace occupation information that allows the robot to check the collision state of tens of millions of samples on-the-fly—the number of which was previously strictly limited by available memory. The hierarchical structure also significantly reduces the time for path searching, hence the robot is able to find feasible motion plans in real-time in extremely constrained environments. The HDRM is theoretically proven to be resolution complete, with a rigorous benchmarking showing that HDRM is robust and computationally fast, compared to classical dynamic roadmap methods and other state-of-the-art planning algorithms. Experiments on the 7 degree-of-freedom KUKA LWR robotic arm integrated with real-time perception of the environment further validate the effectiveness of HDRM in complex environments.



author={Y. Yang and W. Merkt and V. Ivan and Z. Li and S. Vijayakumar},
journal={IEEE Robotics and Automation Letters},
title={HDRM: A Resolution Complete Dynamic Roadmap for Real-Time Motion Planning in Complex Scenes},
keywords={Collision avoidance;Dynamics;Heuristic algorithms;Planning;Probabilistic logic;Real-time systems;Robots;Motion planning;collision avoidance;dynamic roadmap;realtime planning},

Robust Shared Autonomy Demonstrator Overview Video

The University of Leeds has released a video summary on our Robust Shared Autonomy work which won the First Prize for Greatest Potential for Positive Impact at the Robots for Resilient Infrastructure Challenge and was previously also featured on Made In Leeds TV: