Table of Contents
This site provides simple web services for Google App Engine developers.
We think that App Engine is a great product, but it lacks some features. We would like to provide those features to you for free.
Our services are created to extend the possibilities of rapid web development.
- Currently we offer you:
- We're thinking of:
- captcha service (but recaptcha is working fine)
We provide some online examples. The code is available online.
Currently all our services are completely free. The downside is that we don't guarantee anything.
The site is currently under heavy development. That's not event beta phase, it's more like proof of concept. Though we try not to change our APIs.
We don't know how many people will be interested in this services.
If the services becomes a success we plan to offer paid premium accounts with better quality services like:
- guaranteed time of image resizing (the resize will be done in few minutes!)
- lower latency comet service
- forever-lasting channels in comet service
- advanced panel for cron service
We're thinking of charging you few bucks a month for this extra features, but currently we have no idea how much. It depends on many things - for example on the cost of Google App Engine premium accounts.
On Google App Engine you can't easily resize images. PIL is not availble and cpu cycles are a precious asset.
We offer you simple service, that will resize the data for you and upload a resized image to your site.
When our service will resize your image, it's going to call back your application and upload resized image. The service will simply download site you submitted with callback parameter and POST resized data there.
- Limitations:
- resized images are converted to jpeg format, the exception is avatar mode in which the output can be jpeg or gif
- uploaded images can't be bigger than 10 megabytes
- The original and resized images are going to be stored on our server for at most 24 hours.
Here are the examples: Original image:
![]()
action=resize size=50x50:
![]()
action=thumbnail size=50x50:
![]()
action=avatar size=50x50:
![]()
Original image:
![]()
action=resize size=50x50:
![]()
action=thumbnail size=50x50:
![]()
action=avatar size=50x50:
![]()
To see how the service works you can try to use our online form from your browser.
Here is another example, it shows how easy is to use our service from shell using wget:
$ wget 'http://resizer.movq.net/resizer/resize?id=id1234&size=100x100&action=avatar&callback=http://mydomain.com/upload' --post-file=P-38_Lightning_head-on.jpg -q -O - Image uploaded successfully. <br /> source: http://resizer.movq.net/resizer/upload/31iayn6chbohurebz2p732ieu.jpg <br /> destination: http://resizer.movq.net/resizer/upload/31iayn6chbohurebz2p732ieu_100x100.jpg <br /> estimated time: 8 seconds <br />Or if you prefer curl:
$ curl -F 'file=@jP-38_Lightning_head-on.jpg' -F 'id=id1234' -F 'size=100x100' -F 'action=avatar' -F 'callback=http://mydomain.com/upload' 'http://resizer.movq.net/resizer/resize' Image uploaded successfully. <br /> source: http://resizer.movq.net/resizer/upload/rllcqedx4d5nwl70l87o0n0gq.jpg <br /> destination: http://resizer.movq.net/resizer/upload/rllcqedx4d5nwl70l87o0n0gq_100x100.jpg <br /> estimated time: 4 seconds <br />The most important example is the one using Google App Engine:
from google.appengine.api import urlfetch import urllib # remember to create valid callback url query = urllib.urlencode({ 'id': im.secret, 'size': '100x100', 'action': 'thumbnail', 'quality': '85', 'callback': 'http://mydomain.com/gallery/%s/upload' % im.key(), 'file_name': im.filename, }) try: resp = urlfetch.fetch('http://resizer.movq.net/resizer/resize?%s' % query, payload=image_data, method=urlfetch.POST, headers={}, allow_truncated=False) if resp.status_code != 200: logging.error("failed to schedule image resize for img %s, code=%i msg=%.60r" % (im.key(), resp.status_code, resp.content) ) except urlfetch.DownloadError: logging.error("failed to schedule image resize for img %s" % (im.key()) )You also must create a callback url, so resized image is going to be uploaded there. I use simillar code:
# main.py, url dispatcher application = webapp.WSGIApplication( ... ('/gallery/([^/]*)/upload', gallery.views.UploadImage), ... # gallery/views.py class UploadImage(webapp.RequestHandler): def post(self, key): im = db.get(db.Key(key)) if not im: self.error(404) return if self.request.POST['id'] != im.secret: self.error(400) return file_data = self.request.POST['file'].file.read() if self.request.POST['size'] == '100x100': im.thumb = file_data a = 'small' elif self.request.POST['size'] == '500x500': im.small = file_data a = 'full' if im.small and im.thumb: im.published = True im.save() logging.info("%s updated %s" % (im.key(), a) ) self.response.out.write("ok")
You can also see how this code works in our online example gallery (source code).
We try to keep the api minimalistic and straightforward. Where possible, we try to use yaml output format.
The API contains only one http location:
Availble parameters (passed either as GET or POST parameters):
- id : user photo id as a string
- This will be used to identify the image after resizing. This should allow you to identify this image.
- size : designated size of image in format widthxheight
- If you don't want to specify one of those parameters, just insert 0 instead of a valid value like 600x0
- quality : resized image quality, in percents
- The quality of resized image. By default the quality is assumed 85%. Better (bigger) quality means bigger output file.
- bgcolor : the background color
The background color for background in avatar and thumbnail mode. The value can be color description in rgb like #FF0000 for pure red color or transparent string if you want to have transparent background. The transparent background works only for gif images. The default value is white #FFFFFF, for gif images and avatar mode the default is transparent.
renders black background for jpg images.
- action : one of resize|thumbnail|avatar
- resize
scales image in nice way. your image's size is not going to be grater than designated size (can be smaller than one of dimensions), preserving image size ratio.
- Examples:
- action=scale&size=500x500 - the width and the height will never be greater than 500 pixels. If the image is vertical, the width will be 500 but the height will be fitted to blabla the blabla
- action=scale&size=500x0 - the image height will be fitted to .... and width will always be 500px
- thumbnail
scales the image, but without keeping the ratio. The dest image will be always have width and height exactly as you requestd.
- Examples:
- action=thumbnail&size=50x50 - the image will be exactly 50x50, no mater of the source size
- avatar
- does exactly the same as thumbnail except that gif animations are kept.
- callback : user callback url
The resized image will be uploaded there automatically just after the resizing. The resized image is going to be PUSHed on this url, with parameters:
- id
- client's id that you used while uploading the image
- size
- size you requested in format WxH
- action
- action you requested
- file : uploaded file data
- The uploaded file. You can omit this parameter and include the file contents in the body of POST request if you can't use multipart/form-data encoding. In this case all other parameters should be passed as GET parameters.
- file_name : uploadef file name
- If you're sending file data in the body of POST request (when you can't use multipart/form-data) you can explicite set filename. This can be useful if you want to resize gif images.
- Return codes:
- 200
- image resize schedules succesfully
- 400
- one of the parameters has wrong value
Sometimes it's needed to count something at a specific time. Update users couner, garbage collect the data or send emails at specific hour.
This service allows you to request a download of your site at a specific hour.
This feature is simillar to popular cron or at unix command.
- Limitations:
- one uri can be registered only once. If you need to schedule one uri multiple times, please just add some random parameter to the uri, like example.com/send_emails?secret=123dsa&random_param=0.912131314
- To prevent DoS one domain can have only 10 registered jobs.
- Once scheduled download is going to be executed only once. To download a site in regular intervals you have to schedule event each time. This makes this service more simillar to at command, rather than cron.
- You can't register events more than 24 hours in the future.
All time values are in UTC.
Consider this bash session:
$ wget "http://cron.movq.net/cron/schedule" --post-data="time=+1&callback=http://www.google.com/" -q -O - message: Cron wget scheduled successfully. time: 2008-04-30 22:38:45.097494Or if you prefer curl:
$ curl "http://cron.movq.net/cron/schedule" --data "time=+1&callback=http://www.google.com/" message: Cron wget scheduled successfully. time: 2008-04-30 22:40:32.125055This will tell our service to download http://www.google.com next minute.
Or GAE code snippet:
from google.appengine.api import urlfetch import urllib url = 'http://mydomain.com/' query = urllib.urlencode({ 'callback': url, 'time': '+10', }) try: resp = urlfetch.fetch('http://cron.movq.net/cron/schedule', payload=query, method=urlfetch.POST, headers={}, allow_truncated=True) if resp.status_code != 200: logging.error("failed to schedule cron for domain %r" % url ) except urlfetch.DownloadError: logging.error("failed to schedule cron for domain %r" % url )
Only one url is used in this service:
- Parameters:
- callback
- uri to be downloaded at a specific time
- time
- Time that says when to download specified site. This parameter can be in formats:
- +10
- The callback site is going to be downloaded 10 minutes from now.
- +1:10
- The callback site is going to be downloaded one hour and 10 minutes from now, this one is equalivent to +70.
- 21:12
- The callback will occur at specified hour UTC.
- Return codes:
- 200
- request scheduled succesfully
- 400
- the time is in wrong format
Let's assume you want to write chat application on Google App Engine. The simplest way to do that is to regularry poll the server to check if a new event has occurred. This may sound as a reasonable approach, but it isn't.
Let's count: Google offers you 650k http requests daily. Let's say that you're going poll google servers once a second.
650k requests avaible daily / 86k seconds daily = 7 users that are on the chat for full day
Only 7 users on a chat for a day to exploit ther resources is not really a scalable thing. Our comet service aims to overcome these restrictions.
The comet service allows you to easilly use a comet technology benefits, without the need of problematic setting advanced server.
- Limitations:
- you must have your own domain and forward comet.yourdomain.com to our server to defeat browsers security restrictions
- our secirity model is simplified, if you want any kind of authorization - you must do it on your side
- A channel can be created only for 24 hours. After this time (or before) the channel will be destroyed, and you must create a new one.
- Domain configurations known to work:
- your site on mydomain.com, comet site on comet.mydomain.com
- your site on www.mydomain.com, comet site on comet.mydomain.com
- Domain configurations known to not work:
- your site on www.mydomain.com, comet site on comet.www.mydomain.com
- Browsers:
Browser Comet engine Loading bar hidden Automatic reload (when server dies) Multiple comet sources Status Firefox 2 xhr_stream n/a yes yes supported Safari/Webkit 3.1 xhr_stream partially yes yes supported Opera 9.2 sse n/a no (reloads after keepalive timeout) yes supported Internet Explorer 7 htmlfile yes yes not tested supported Internet Explorer 6 htmlfile yes yes not tested not tested Internet Explorer 5.5 htmlfile yes yes not tested not tested Konqueror iframe yes yes not tested not tested We're considering supporting flash-based comet engine.
We have scalability in mind, but currently we don't have enough hardware to guarntee full scalability. On current hardware we're able to support few hundred simultaneus users. We're prepared to extend our infractructure in few days if there will be demand for it.
- Example applications (source code):
- simple chat application (many-many)
- real time information about views counter changes (one-many)
Our job is to provide service for you. The browser side is your problem.
Usually comet connections are scheduled on the same server as the rest of the site. In our case, the site is served from your (or google's) servers, and comet is served from our servers.
It's quite hackish to make browsers accept this, we need to use cross-domain-comet connections.
It can be described:
your html (mydomain.com) -> iframe on your domain, on our server (comet.mydomain.com/_/iframe.html) -> iframe/xmlHttpRequest to our comet service
But to simplify the programming process we created some helpers, which should help you.
Before loading our scripts please insert something like this to your head section:
<script type="text/javascript" src="http://comet.${servicesurl}/_/comet.js"></script>If possible this files should be hosted on your site for speed and security.
The interface contains:
This is the file with all our javascript scripts. It's less than 500 lines long, so you can try to read the source.
- Constants:
- comet_restart_timeout
- How long to wait befor reconnecting when the comet connection is broken (in milliseconds, 5 seconds by default.)
- comet_keepalive_timeout
- How long can a connection last without any messages passing. The connection will be recreated after this time unless some event occurs. (in milliseconds, 76 seconds by default, this should be okay for 60-seconds keepalive timer)
- The important functions:
- comet_connection(url, user_callback)
This is the main function that creates comet connection. It also reconnects if the keepalive is not received before comet_keepalive_timeout.
This function is probably useless for you, because it assumes that the comet server is in the same domain as your site.
- Usage example:
<script> function callback(data){ alert('got event: ' + data); } /* comet is on the same server as this site */ comet_connection('/path/to/comet/listen', callback); </script>- Parameters:
- url
- This can be a string that is valid url to long comet connection. It can also be a function which returns url. The function will be executed every time the url is needed, that means before the first connection and before every reconnection. The url must be local.
- user_callback
- Your callback function, it's going to be executed every time the data will appear on connection.
- comet_schedule_crossdomain(iframe_uri, comet_uri, user_callback)
The function is a wrapper for comet_connection, which does exactly the same as previous function except that it works cross-domain. It loads the iframe_uri into an iframe, sets document.domain and passes parameters.
You should set domain comet.yourdomain.com to point to our servers.
Before running this function make sure that document.domain on your site is set to good value.
- Usage example:
<script> /* yoursubdom.yourdomain.com -> yourdomain.com */ document.domain = extract_xss_domain(document.domain); function callback(data){ alert('got event: ' + data); } /* comet.yourdomain.com points to our server */ comet_schedule_crossdomain('http://comet.yourdomain.com/_/iframe.html', '/comet/' + channel_id + '/listen', callback); </script>- For our service the data received by your callback can be:
- keepalive_message
- If no event occurs for a period of time. server sends keepalive message with the message you specified when creating the channel. By default it's ping string.
- connect_message username
- Just after connecting, the server can send you last 10 messages. After this events the server sends connect message you specified when creating the channel. By default its connect string. After the connect string the current set username is appended. If you haven't specified username while connecting to listen service, it will be set to random value.
- 'destroyed' string
- When the channel is destroied this string is received by all the listeners. You must create new channel and receive the new channel_id in all browsers.
- event_id author message
Normal event is delivered in this format.
- Details:
- event_id
- Numeric value that identifies current message.
- author admin|guest
- If the channel is public everyone can send messages. The messages that used valid event_key when sending event are marked as admin, all others as guest. If the channel is not public, only admin messages will occurr.
- message
- The sent message. It's not escaped, so please be sure to quote it before viewing.
More specific example of user callback function:
<script> var max_id = 0; function comet_callback(data){ var a = data.split(' '); var id = a[0]; var author = a[1]; /* system messages */ if(id == 'connected'){ username = author; return; } if(id == 'ping'){ return; } if(id == 'destroyed'){ // this channel_id is no longer valid return; } id = Number(id); /* ignore duplicate events */ if(id <= max_id) return; max_id = id; /* user event */ var message = a.slice(2).join(' '); alert('event_id:' + id + ' author:' + author + ' message:' + message); } </script>- ajax_crossdomain(iframe_uri, method, ajax_uri, user_callback, post_data)
Sometimes you could need to send normal ajax requests to our comet service. This can't be done straightforward - domains don't match. To do this you can use this wraper.
For example common case is to send events directly from browser to our server, rather than to your site. This code can help you:
- Usage example:
<script> /* yoursubdom.yourdomain.com -> yourdomain.com */ document.domain = extract_xss_domain(document.domain); function cb(o){ /* the parameter is the connection object from YUI see: http://developer.yahoo.com/yui/examples/connection/post.html */ if(o.status != 200) alert('sending event failed'); } ajax_crossdomain('http://comet.yourdomain.com/_/crossajax.html', 'POST', '/comet/' + channel_id + '/event', cb, 'data=' + escape('this is the message!') ); </script>
The service is composed from this api. Where possible, we try to use yaml output format.
- Parameters:
- event_callback_uri
- send received event data not only to the clients, but also to this url (which is short connection) Will be send using POST method and the data will be in parameter event.
- keepalive_time
- How often to send keepalive data. 60 seconds by default, 120 seconds max.
- keepalive_message
- Message to be send on keepalive event. By default this is 'ping' string.
- connect_message
- Data to be sent to client just after connecting. By default this is 'connected', to look like keepalive message.
- public
- Is the chalnnel public? anyone can send events? should be False|True.
- Return codes:
- 200
- new channel created successfully
Returns identifier to the new channel and security key for sending events and another one for destoring the chanenel
informaton about the channel. users listening, how many events/connection send since createion, createion time, latency problems etc.
- Return codes:
- 200
- information page got sucessfuly
- 404
- channel doesn't exist
will return 200 status code if the channel exists or 404 else.
long-polling connection from the channel (used in javascript on the browsers side)
- Parameters:
- last_eventid
- eventid of the last message received by the listener. If this parameter is not specified last 10 events are repeated.
- username
- username to identify this connection. Can be empty.
- Comet specific parameters:
- transport
- transport type, could be one of basic|iframe|htmlfile|server_sent_events|sse|xhr|xhr_stream|xhr_multipart. See Orbited project for details.
- callback
- callback function. Used for transports htmlfile|iframe
- domain
- currently ignored.
- Return codes:
- 200
- everything okay, this connection is long one.
- 404
- channel doesn't exist
You rather won't use this uri directly. Please see our javascript interface function comet_schedule_crossdomain.
sends an event to every listener
- Parameters:
- data
- data to be send (you're responsible for escaping the data if needed). This data shouldn't contain more than 256 bytes for one event.
- event_key
- secret key needed to send data (received when creating channel) if channel is not public. if it's - no event_key is needed
- Return codes:
- 200
- event send
- 204
- data is empty
- 403
- event_key is not valid, you're not allowed to send events
- 404
- channel doesn't exist
If you need to send events from browser to channel, please consider our javascript method ajax_crossdomain. Remember about setting the channel to be public or pass valid event_key to the browsers.
- Parameters:
- destroy_key
- must be a valid security key for destroying the channel
destroies the channel.
the channel is going to be destroyed automatically about 24 hours after last received event.
- Return codes:
- 200
- channel destroyed
- 403
- destroy_key is not valid, you're not allowed to destroy this channel
- 404
- channel doesn't exist