↓ Archives ↓

Category → rdesktop

Capturing the Screen or Video of Your Virtual Machines Using X, Vnc, Rdp or Native ways

With this blogpost we continue the previous investigation to interact with a virtual machine using X, VNC , RDP or native ways with a virtual machine. This time instead of sending keystrokes we are looking for capture screenshots or even capture the complete session as a video.

Interacting with X-Windows

Capturing a screenshot:

Grabbing a screen of an X-Windows session is easy: in order to grab the screen on an X-session on :1 issue the following command:

$ import -window root -display :1 screenshot.png

Recording a video:

I got the idea of using the ffmpeg command for capturing the X-session from

http://gasubasu.com/2009/05/06/flash2video/

So I installed ffmpeg using macports

$ sudo port install ffmpeg

When I executed it, it complained about Unknown input format: 'x11grab'

$ ffmpeg -f x11grab -vc x264 -s vga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 29 2010 16:56:51 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --prefix=/opt/local --enable-gpl --enable-postproc --enable-swscale --enable-avfilter --enable-avfilter-lavf --enable-libmp3lame --enable-libvorbis --enable-libtheora --enable-libdirac --enable-libschroedinger --enable-libfaac --enable-libfaad --enable-libxvid --enable-libx264 --enable-libvpx --enable-libspeex --enable-nonfree --mandir=/opt/local/share/man --enable-shared --enable-pthreads --disable-indevs --cc=/usr/bin/gcc-4.2 --arch=x86_64
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     1.11. 0 /  1.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
Unknown input format: 'x11grab'

There was no variant or option for macports to have ffmpeg take this flag for compiling. So I downloaded the sourcecode from the ffmpeg repository:

$ wget http://www.ffmpeg.org/releases/ffmpeg-0.6.tar.gz
$ cd ffmpeg-0.6
$  ./configure --enable-x11grab --enable-gpl

Although I had specified the --enable-x11grab option, ffmpeg still complained about the Unknown input format.

I got some clues here and here. And after a look at the the configure file, I found that in order to work, it required the Xfixes and Xext libraries:

enabled x11grab                         &&
check_header X11/Xlib.h                 &&
check_header X11/extensions/XShm.h      &&
check_header X11/extensions/Xfixes.h    &&
check_func XOpenDisplay -lX11           &&
check_func XShmCreateImage -lX11 -lXext &&
check_func XFixesGetCursorImage -lX11 -lXext -lXfixes

So after installing those libraries:

$ sudo port install xorg-libXext
$ sudo port install xorg-libXfixes
$  ./configure --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags="-I/opt/local/include -I/opt/local/include/X11"  --extra-ldflags=-L/opt/local/lib

....
Enabled indevs:
x11_grab_device
...

Now with the input x11_grab_device available, I was all set to record a running session.

$ ./ffmpeg -f x11grab -vc x264  -s xga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 30 2010 09:11:02 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags='-I/opt/local/include -I/opt/local/include/X11' --extra-ldflags=-L/opt/local/lib --disable-shared
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libswscale     0.11. 0 /  0.11. 0
[x11grab @ 0x10100b000]device: :1.0 -> display: :1.0 x: 0 y: 0 width: 1024 height: 768
[x11grab @ 0x10100b000]shared memory extension  found
[x11grab @ 0x10100b000]Estimating duration from bitrate, this may be inaccurate
Input #0, x11grab, from ':1.0':
  Duration: N/A, start: 1283153847.281641, bitrate: 754974 kb/s
    Stream #0.0: Video: rawvideo, bgra, 1024x768, 754974 kb/s, 30 tbr, 1000k tbn, 30 tbc
File 'session-recording.avi' already exists. Overwrite ? [y/N] y
Output #0, avi, to 'session-recording.avi':
  Metadata:
    ISFT            : Lavf52.64.2
    Stream #0.0: Video: mpeg4, yuv420p, 1024x768, q=2-31, 2000 kb/s, 30 tbn, 30 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop encoding
frame=  310 fps= 30 q=2.0 Lsize=    2810kB time=11.83 bitrate=1945.1kbits/s dup=0 drop=21    
video:2796kB audio:0kB global headers:0kB muxing overhead 0.502798%

This worked fine! But the recording only captured 1024x768 (flag = xga), when I tried to have ffmpeg capture a larger screen 1600x1024 (flag wsxga), it told me an error about Can't get shared memory . No clue why. For now, the xga option is enough for me.

./ffmpeg -f x11grab -vc x264  -s wsxga -r 30 -b 2000k -g 300 -i :1.0 session-recording.avi
FFmpeg version 0.6, Copyright (c) 2000-2010 the FFmpeg developers
  built on Aug 30 2010 09:11:02 with gcc 4.2.1 (Apple Inc. build 5664)
  configuration: --enable-x11grab --enable-gpl --enable-nonfree --extra-cflags='-I/opt/local/include -I/opt/local/include/X11' --extra-ldflags=-L/opt/local/lib --disable-shared
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libswscale     0.11. 0 /  0.11. 0
[x11grab @ 0x10100b000]device: :1.0 -> display: :1.0 x: 0 y: 0 width: 1600 height: 1024
[x11grab @ 0x10100b000]shared memory extension  found
[x11grab @ 0x10100b000]Fatal: Can't get shared memory!

Interacting with VNC

Capturing a screenshot:

I found two projects that touted the ability to capture the screen of a VNC session:

None of them really worked for me. So I resorted back to using Xvfb as an intermediate:

$ vncpasswd mypasswordfile
$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard
$ DISPLAY=:1 rdesktop -u username -p password -d domain remotehost
$ import -window root -display :1 screenshot-rdp.png

Recording a video:

To record an vnc session to flash I found the following pointers:

I went for the C version of vnc2swf :

$ wget http://www.unixuser.org/~euske/vnc2swf/vnc2swf-0.5.0.tar.gz
$ cd vnc2swf-0.5.0
$  LDFLAGS=-L/usr/X11R6/lib  ./configure 
$  LDFLAGS=-L/usr/X11R6/lib  make
$ ./vnc2swf
usage: ./vnc2swf [<options>] <out.swf> <host>:<display#>
       ./vnc2swf [<options>] <out.swf> -listen [<display#>]

<options> are standard Xt options, or:
              -shared
              -viewonly
              -fullscreen
              -passwd <passwd-file>
              -noauto
              -encodings <encoding-list> (e.g. "raw copyrect")
              -geometry <geometry>
              -bgr233
              -owncmap
              -truecolour
              -depth <depth>
              -framerate <frames-per-sec>
              -startrecording
              -nowindow
              -nostatus
              -clippinggeometry <geometry>

Àn alternative is to ffmpeg as explained in the X-session, with vncviewer displayed in Xvfb session


Interacting with RDP

Capturing a screenshot:

I found 0 tools to capture a screenshot directly from an RDP (at least a tool on Linux or MacosX). While poking around I found a reference to VNC inside the code of Rdesktop: there was mention of rdp2vnc

$ ./configure --help |grep vnc
  --with-libvncserver-config=CMD  use CMD as libvncserver-config
  --with-libvncserver     make rdp2vnc

I almost got it to work by checking out the latest svn

$ svn co https://rdesktop.svn.sourceforge.net/svnroot/rdesktop rdesktop
$ cd rdesktop

It required the installation of LibVNCServer library - http://libvncserver.sourceforge.net/

./configure --prefix /opt
make
make install
$ cd rdesktop
$ ./configure  --with-libvncserver  --x-includes=/opt/local/include/ --x-libraries=/opt/local/lib --with-libvncserver-config

But in the end - nothing and I was facing a Compile problem

So of to the X-session fallback using rdesktop and xvfb:

$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard
$ DISPLAY=:1 vncviewer -FullColor --Passwordfile mypasswordfile localhost
$ import -window root -display :1 screenshot-vnc.png

Recording a video:

The only way (I found) is to capture the RDP session as a video is to use ffmpeg as explained in the X-session, with rdesktop displayed in Xvfb session.


Virtual Server helpers:

Capturing screenshots with Vmware Esx

The Vsphere API allows the creation of a task by using CreateScreenshot_Task() command. I used the VMware VI (vSphere) Java API - http://vijava.sourceforge.net/ and coded an example to create the screenshot. The file itself is saved by vmware in the datastore root, so you need something like scp to grab it from there.

import java.net.URL;
import com.vmware.vim25.*;
import com.vmware.vim25.mo.*;

public class takescreenshot 
{
    public static void main(String[] args) throws Exception
    {
        String url="https://localhost/sdk"; 
        String username="root"; 
        String password="thepassword"
        long start = System.currentTimeMillis();
        ServiceInstance si = new ServiceInstance(new URL(url), username, password, true);
        long end = System.currentTimeMillis();
        System.out.println("time taken:" + (end-start));
        Folder rootFolder = si.getRootFolder();
        String name = rootFolder.getName();
        System.out.println("root:" + name);
        ManagedEntity[] mes = new InventoryNavigator(rootFolder).searchManagedEntities("VirtualMachine");
        if(mes==null || mes.length ==0)
        {
            return;
        }
        
        VirtualMachine vm = (VirtualMachine) mes[0]; 
        Task screentask=vm.CreateScreenshot_Task();
        si.getServerConnection().logout();
    }
}

For an example using the perl API have a look at the script at http://communities.vmware.com/docs/DOC-10497

Screencapture and recording a video in Virtualbox

I found that virtualbox itself has a way to capture the screen (--capture) and to record the session to file. Internally it uses ffmpeg. I wasn't able to run it on my mac, as it is not supported on that platform, but it might work for you.

$ VBoxVRDP --help
Oracle VM VirtualBox Headless Interface 3.2.8
(C) 2008-2010 Oracle Corporation
All rights reserved.

Usage:
   -s, -startvm, --startvm <name|uuid>   Start given VM (required argument)
   -v, -vrdp, --vrdp on|off|config       Enable (default) or disable the VRDP
                                         server or don't change the setting
   -p, -vrdpport, --vrdpport <ports>     Comma-separated list of ports the VRDP
                                         server can bind to. Use a dash between
                                         two port numbers to specify a range
   -a, -vrdpaddress, --vrdpaddress <ip>  Interface IP the VRDP will bind to 
   -c, -capture, --capture               Record the VM screen output to a file
   -w, --width                           Frame width when recording
   -h, --height                          Frame height when recording
   -r, --bitrate                         Recording bit rate when recording
   -f, --filename                        File name when recording.  The codec
                                         used will be chosen based on the
                                         file extension

Sending Keystrokes to Your (Virtual) Machines using X, Vnc , Rdp or Native ways

Options overview

The most common way to interact with a virtual machine is by remote login via ssh. This blogpost is about a different way of interaction: it will show you how to send keystrokes (or mouse) directly to the remote screen of the machine. This can be used for instance for kickstarting a machine before the network is up (typing linux), or automating things that require screen interaction.

In general three types of remote screen sessions exist:

  • VNC: is a graphical desktop sharing system that uses the RFB protocol
  • RDP: A proprietary developed by Microsoft
  • X-session: based on the X Window System (commonly X or X11)

Most virtualization solutions allow you to activate one of these remote session options:


Keycodes versus keys

Before we start of with the different solutions, I would like to point out that sending keystrokes to these remote sessions is not the same as printing a character to the screen.

Keyboards and for that matter virtual keyboards interact with scancodes : every key on the keyboard is assigned a scancode. This has a few consequences:

  • An uppercase letter is a combination of a scancode for Shift and a scancode for the letter
  • The same scancode on a different keyboard layout can cause a different letter to be send: f.i. scancode 10 (hex) generate a 'Q' on a Qwerty keyboard but 'A' on an Azerty keyboard

Most of the tools assume the use of a US Layout keyboard. A good overview of the different scancodes can be found at http://www.win.tue.nl/~aeb/linux/kbd/scancodes-1.html


Interacting with X-Windows

Before VNC and RDP existed , people already had the option of using an X-client to an X-server.

P.S. I know you are probably looking for RDP or VNC interaction, but this description will come in handy as we will use this through Xvfb.

I found two tools that help sending keystrokes:

I couldn't get xte to compile on my mac but xdotool is conveniently available in macports.

What is xdotool? This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11's XTEST extension and other Xlib functions. Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

$ sudo port install xdotool
--->  Computing dependencies for xdotool
--->  Fetching xdotool
--->  Verifying checksum(s) for xdotool
--->  Extracting xdotool
--->  Applying patches to xdotool
--->  Configuring xdotool
--->  Building xdotool
--->  Staging xdotool into destroot
--->  Installing xdotool @2.20100818.3004_0
--->  Activating xdotool @2.20100818.3004_0

To use xdotool (and avoid the error message "Error: XTEST extension unavailable
on '(null)'") you need to enable the XTEST extension. If you're using Apple's
X11.app, the command to do so is: 

defaults write org.x.X11 enable_test_extensions -boolean true 

If you're using the MacPorts X11.app, use: 

defaults write org.macports.X11 enable_test_extensions -boolean true 

This only needs to be done once.
--->  Cleaning xdotool

Sending the key 'a' to your X-sessions

$ xdotool key 'a'

The tool has a lot of ways of interacting with the windows, and for our purposes it has a option to send a key and even mouse .

$ xdotool
Usage: xdotool <cmd> <args>
Available commands:
  getactivewindow
  getwindowfocus
  getwindowname
  getwindowpid
  search
  selectwindow
  help
  version
  click
  getmouselocation
  key
  keydown
  keyup
  mousedown
  mousemove
  mousemove_relative
  mouseup
  type
  windowactivate
  windowfocus
  windowmap
  windowmove
  windowraise
  windowsize
  windowunmap
  windowreparent
  windowkill
  set_window
  behave
  set_num_desktops
  get_num_desktops
  set_desktop
  get_desktop
  set_desktop_for_window
  get_desktop_for_window
  get_desktop_viewport
  set_desktop_viewport

Interacting with VNC

Using Ruby-vnc

The easiest way I found to interact with a VNC session is by using the excellent ruby-vnc library http://code.google.com/p/ruby-vnc/

The example on the website gives you a good idea on how it works:

# launch xclock on localhost. note that there is an xterm in the top-left
Net::VNC.open 'localhost:0', :shared => true, :password => 'mypass' do |vnc|
  vnc.pointer_move 10, 10
  vnc.type 'xclock'
  vnc.key_press :return
end

Using vncviewer and xvfb

If you are not that confident with ruby, there is another option. Xvfb allows you to create a virtual screen on which you can interact. Together with the vncviewer (X-client) we can use it with xdotool to interact with a vnc session.

The first step is to create a vnc password file, as with vncviewer the password can not be supplied on the commandline.

Create password file name 'mypasswordfile'

$ vncpasswd mypasswordfile
Password:
Verify:

Start Xfvb screen (:1 means another display and 24 is the colordepth)

$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard

Now we can start vncviewer (that logs in automatically) within the Virtual Frame buffer

$ DISPLAY=:1 vncviewer -FullColor --Passwordfile mypasswordfile localhost

Now we are back to using xdotool to interact with the session.


Interacting with RDP

Using properjavardp

Properjava rdp - http://properjavardp.sourceforge.net/ is a full implementation of the RDP protocol in Java. It doesn't seem to be maintained and several 'forks' are available.

I used the original source to get it running:

  • Import the src directory in a java project
  • Add the src files like RdpPacket_Localised for the src-1.4 directory
  • Add the jar files to the project (log4j.jar, ..)

The following code will give you an idea on how you can use it

package net.propero.rdp;

import java.net.InetAddress;
import net.propero.rdp.rdp5.Rdp5;
import net.propero.rdp.rdp5.VChannels;

public class SendKeysRdp {
    public static void main(String[] args) {
        int logonflags = Rdp.RDP_LOGON_NORMAL;
        Rdp5 RdpLayer = null;
    
        VChannels channels = new VChannels();
        RdpLayer = new Rdp5(channels);
        Common.rdp = RdpLayer;        
        try {
/*
 *                   RdpLayer.connect(Options.username, InetAddress
                         .getByName(server), logonflags, Options.domain,
                         Options.password, Options.command,
                         Options.directory);         
 */
            RdpLayer.connect("Username", InetAddress.getByName("192.168.2.30") , logonflags, "Domain","Command", "","Directory");
            RdpLayer.sendInput(Input.getTime(), Input.RDP_INPUT_SCANCODE, Input.RDP_KEYPRESS, 0x1f, 0);

        } catch(Exception e){
            System.out.println(e.toString());
        }
    }
}

Using rdesktop and Xvfb

$ sudo port install rdesktop
$ Xvfb  :1 -screen 0 1024x768x24 -fbdir /var/tmp/
(EE) AIGLX error: dlopen of /usr/X11/lib/dri/swrast_dri.so failed (dlopen(/usr/X11/lib/dri/swrast_dri.so, 5): image not found)
(EE) GLX: could not load software renderer
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: No components provided for device Virtual core keyboard
$ DISPLAY=:1 rdesktop -u <username> -p <password> -d <domain> remotehost

Directly from C code re-using code from rdesktop

I found a link to an xrdp overflow tool that seems to use the rdp_send_scancode function to send the keystroke.

http://webcache.googleusercontent.com/search?q=cache:GIucABEgkT8J:packetstormsecurity.org/0904-exploits/xrdp-overflow.txt+rdp_send_scancode&cd=7&hl=nl&ct=clnk&gl=be&client=firefox-a

Another cool trick I picked up, was the way to launch commands using rdesktop :

http://www.singularity.be/2008/03/using-rdesktop-to-script-windows.html


Virtualbox option

Virtualbox provides a way to send keyboard scancodes directly using it's excellent API. The option keyboardputscancode allows to specify the hex code of the scancode. You can also send multiple keycodes after each other. In my experience this doesn't work well for a long sequence. It seems the buffer it limited and your best option is to send the different keycodes in multiple calls to the command line.

VBoxManage controlvm        <uuid>|<name>
                            pause|resume|reset|poweroff|savestate|
                            acpipowerbutton|acpisleepbutton|
                            keyboardputscancode <hex> [<hex> ...]|

Using a non-headless solution

In order to automate visual tasks, there exist a lot of macro/recording tools, the downside of them is that they need an actual display to run. When scripting you are mostly looking for headless solutions though. The disadvantage of headless solutions is that they are cumbersome to create as you have to move the pointer and the keys at the exact spot.

Sikuli

When you are interacting with a session you as a person do a lot more then just typing things. You yourself also locate which window to focus, what button to click.

Sikuli is a great tool that tries to help you in those tasks. It uses image recognition tools to find the correct place on the screen. Think of it a visual scripting language for screen interaction. An example script looks like this:

It allows you use these commands within java and it seems that someone is working on replacing the java.awt.robot to a vnc version

More background can be found at:

The function type() simulates keyboard typing just as a user types text in a application. However, type() doesn't work for different keyboard layouts other than QWERTY, such as DVORAK. We provide a workaround paste() since 0.9.7 (20100127). The function paste() transfers text through system's clipboard, which is fully independent of keyboard layouts. A sample usage that paste "network" into a search box is shown as follows.

Other options

For completeness I provide different options I found useful

$ osascript -e 'tell application "System Events" to keystroke "LOGIN_NAME"'; \
$ osascript -e 'tell application "System Events" to keystroke tab'; \
$ osascript -e 'tell application "System Events" to delay 0.5'; \
$ osascript -e 'tell application "System Events" to keystroke "PASSWORDHERE"'; \
$ osascript -e 'tell application "System Events" to delay 0.5'; \
$ osascript -e 'tell application "System Events" to keystroke return'

Replace LOGIN_NAME and PASSWORD with the proper values
Run as "root" user