Robotics Laboratory

Department of Computer Science | Iowa State University

Manipulation of a Manual Screwdriver

Despite decades of steady research progress, the robotic hand is still far behind the human hand in terms of dexterity and versatility. A milestone in this quest for human-level performance will be possessing the skills of manipulating hand tools, for their non-trivial geometries and for the intricacies of controlling their contact-based interactions with objects, which are often the targets of manipulation.

This work investigates screwdriving by a robotic arm/hand pair, dealing with the chain of contacts connecting the substrate, screw, screwdriver, and fingertips. Considering rolling contacts and gaits of the fingers, our force control scheme is derived through backward chaining to leverage the dynamics of the screwdriver and arm/hand. To maintain the fastening effort, estimations are carried out sequentially for the screwdriver's pose, via optimization under visual and kinematic constraints, and for its applied wrench on the screw, via solution drawing upon dynamics. This wrench, adjusted based on position/force feedback, is mapped through the grasp matrix to the desired fingertip forces, which are then used for computing torques to be exerted by the arm and hand to close the loop.

1. Mounting of the Screwdriver

We assume that the screw 1) is pre-installed in a tapped hole; 2) has approximately known location; 3) has unknown drive orientation.

The mounting starts with the tip touchdown onto the screw head while maintaining a small tilting angle \(\eta\) via position control. When the tip contacts the head, the F/T sensor readings will increase suddenly because of the impact. To soften it, we switch from position control to impedance control in the contact normal direction.

Then the tip slides along two orthogonal paths with the same tilting angle \(\eta\) via position/force control. Tip sliding ends in one of two situations: 1) it reaches one of the straight edges bounding the drive; and 2) it reaches the circular boundary of the head. In both situations, the contact will experience a sudden change to be captured by the F/T sensor at the end-effector. We distinguish these two cases by continuing the sliding and checking if a second contact is encoutered.


Once the tip is inside the drive with contacts established at both edges of the drive, it is inserted in a sequence of three rotations. First, (a-b) rotate the tip while maintaining its contact with the slot edge via force control until it falls into the drive. Then, (c) soften the impact via impedance control and rotate the tip to align its edge with the drive’s bottom. Finally, (d) constrain the tip via a third rotation about its axis.

2. Modeling Force/Torque on the Screw from the Substrate

Consider the screw thread as the trajectory of a triangular element rotating and translating along the screw axis simultaneously. The three vertices of this triangle move along separate helices: the crest curve \({\bf c}(\phi) = \left( c_1 \cos \phi, -c_1 \sin\phi, \frac{\phi}{2\pi}p\right)^\top,\) and the upper/lower root curves \({\bf r}_{u/l}(\phi) = \left( c_2 \cos \phi, -c_2 \sin\phi, \frac{\phi}{2\pi}p \pm \frac{p}{2}\right)^\top,\) where \(\phi\) gives the amount of the rotation along the helix, \(c_1\) is the major radius (distance from the crest to the screw/hole axis), \(c_2\) is the minor radius (distance from the root), and \(p\) denotes the pitch.

As we walk along the vertices on both threads from top to bottom, we can check if the screw penetrates into the wall or not at each vertex \(v_i\) by comparing it with the point on the other thread at the same height. With all the vertices of the penetration region \(\Omega\) known, we can calculate its area \(A\). Then the forces are estimated based on the area using a simple spring model \(F=kA\), where \(k\) is a stiffness constant. Finally, we integrate all the forces and torques for all cross sections over \(\phi = [0, 2\pi)\) to get the force and torque exerted on the screw by the threads in the hole.

3. Screwdriving by the Hand-Arm System


Architecture of the system of robotic screwdriving

3.1 Estimating the Screwdriver Pose From Vision and Hand Kinematics

(a) Experiment image. (b) Initial (red) and refined (blue) pose estimates of the screwdriver in (a).

In a dexterous manipulation task, pose estimation solely relying on vision cannot provide the needed accuracy. Here we propose a method that takes an estimated tool pose from vision and polishes it under contact constraints imposed by fingertips. It is solved as an optimization problem for a screwdriver pose that is closest to the visual estimation, while satisfying the kinematic constraints imposed by the contacts as follows:

(1) The screwdriver's tip edge is in contact with the bottom of the screw's drive.

(2) Each finger contact "agrees with" the corresponding contact on the handle in location and geometry.

(3) Each finger contact on the handle does not move significantly from its position at the previous time instant.

3.2 Estimating the Force From the Substrate

To control the torquing wrench applied by the screwdriver to the screw, we need to estimate the force/toque transmitted from the substrate through the screw to the screwdriver tip.

Denote by \({\bf \nu}\) the linear and angular velocity of the tool frame (located at the center of the tip's edge) as expressed in the frame itself. The Newton-Euler equations governing the screwdriver dynamics are described in this frame as

$${\bf F}_s + {\bf F}_h = M \dot{\bf \nu} + {\bf C}$$

where \({\bf F}_s = ({\bf f}^\top_s, {\mathbf \tau}^\top_s)^\top\) is the force/torque exerted by the screw, and \({\bf F}_h = ({\bf f}^\top_h, {\mathbf \tau}^\top_h)^\top\) is the resultant force/torque applied on the screwdriver by the robotic hand. \({\bf F}_h\) is obtained by mapping the contact forces exerted and sensed by the fingertips on the screwdriver's handle using the grasp matrix determined by the hand configuration.

Among the 6 unknowns in \({\bf F}_s\) and 6 more in \(\dot{\bf \nu}\), half of them can be eliminated based on the following observation: the contact between the screwdriver's tip and the screw's drive acts most of the time like a hinge with one DOF that allows the tip to tilt forward or backward. Thus, we can rewrite the euqations as

$$({\bf f}^\top_s, 0, {\tau_{s,y}}, k_s \theta)^\top + {\bf F}_h = M \left(0, 0, \frac{p}{2\pi} {\dot{\omega}_z} , {\dot{\omega}_{x}}, 0,{\dot{\omega}_{z}}\right)^\top + {\bf C} \text{.} $$

Solving the above linear equation, we will use the obtained \({\bf F}_s = ({\bf f}_s^\top, 0, \tau_{s, y}, k_s\theta)^\top\) later for control of the screwdriver.

3.3 Constructing the Desired Hand Wrench

The control of the screwdriver, conducted within the tool frame, aims at finding a desired wrench \({\bf F}_{hd}\) to be received from the hand. We utilize the dynamics of the screwdriver by replacing \({\bf F}_h\) with \({\bf F}_{hd}\), and \({\bf F}_s\) and \(\dot{\bf \nu}\) respectively with control servos \({\bf F}_{sc}\) and \({\bf \alpha}\), which yields

$${\bf F}_{sc} + {\bf F}_{hd} = M {\bf \alpha} + {\bf C} \text{.}$$

This equation describs the "desired" dynamics of the screwdriver which is realized via error compensations.

In the rotational directions, we construct Proportional-Integral-Differential (PID) position control servos in \({\bf \alpha}\). The servos take the pose estimation as feedback to control the fastening speed (about \(z\)-axis) and correct the tilting error (about \(x\)- and \(y\)-axies).

In the translational directions, we construct force control servos in \({\bf F}_{sc}\). They take sensed force at fingertips to maintain the desired downward force (along \(z\)-axis) and prevent excessive forces from causing the tip to slide (along \(x\)- and \(y\)-axes).

The desired wrench \({\bf F}_{hd}\) is then obtained by solving the above equation.

3.4 Finger Force Control

Control of the fingers aims at applying an appropriate force at each contact in order to realize \(^w{\bf F}_d\) so rolling continues. In order to keep the contact force of each finger inside the corresponding friction cone, we need to add an internal force in the null space of the grasp matrix \(G\). The desired contact forces are modified as

$${^w{\bf F}_m} = G^\dagger \text{diag}[R, R] {\bf F}_{hd} + (I - G^\dagger G){^w{\bf F}_0} \text{,}$$

where in the second summand, a vector \(^w{\bf F}_0\) is projected to the null space of the grasp matrix.

Then we apply force control at each fingertip to realize \({^w{\bf F}_m}\). The dynamics of the arm-hand system is described in the world frame as

$${\bf \tau} = M_r\ddot{\bf q} + C_r\dot{\bf q} + {\bf N} + J^{\top} {^w{\bf F}} \text{.}$$

In the above dynamics, \(\ddot{\bf q}\) is obtained from projecting the desired acceleration of the screwdriver to the configuration space. \({^w{\bf F}}\) is placed under force control such that it regulates the error between the desired contact forces \({^w{\bf F}_m}\) and the actual ones from the sensors at the fingertips.

3.5 Finger Gaiting

Finger rolling can only rotate the screwdriver for a small angle, and rounds of rollings are connected by finger gaiting. We let the thumb (TH) and middle finger (MF) perform the fastening task. Whenever a finger joint reaches its limit or the contact on a fingertip reaches the boundary of its tip surface, both TH and MF are relocated properly for the next round of screwdriving via finger gaits. Before either of them breaks contact, the index finger (IF) and ring finger (RF) are added on the screwdriver's handle to keep it in balance.


The screwdriver handle is grasped by the TH and MF in (a), and then the IF and RF are added in (b). The first pair is removed in (c) and then relocated in (d). The second pair is then removed and, after the first pair has performed some screwdriving in (e), added back in (f) for the next round. (g) Before and (h) after three rounds of finger rolling and gaiting.



For more information, we refer to the following submission:

Last updated on Oct 01, 2023.