Talking to IMAP services can be a source of frustration especially when each service implements parts of the protocol or add some secret ingredients to the sauce. One of the most popular services, as we all know is Gmail.

XOAuth

Gmail adds a lot of nice touches to the mail business. One of the latest additions is the use of XOAuth in order to authenticate against the service without any need of username and password. This can be especially handy if your app your app should read or just parse the headers of the users' emails but you do not want to save passwords, have extra UI for this purpose or let the user revoke the access at will. You can easily use xoauth in your webapp today using this small library.

Example code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def connect_to_gmail(CREDENTIALS, email, oauth_token, oauth_token_secret):
    """
    Call this function to get an authenticated IMAP connection
    """
    consumer = OAuthEntity(CREDENTIALS[0], CREDENTIALS[1])
    access_token = OAuthEntity(oauth_token, oauth_token_secret)
    xoauth_string = GenerateXOauthString(
      consumer, access_token, email, 'imap',
      None, str(random.randrange(2**64 - 1)), str(int(time.time())))

    # connect to imap
    imap_conn = imaplib.IMAP4_SSL('imap.googlemail.com')
    #imap_conn.debug = 3
    imap_conn.authenticate('XOAUTH', lambda x: xoauth_string)
    imap_conn.select('INBOX')

Searching

By default when you use imap.select() you will be able to search within the INBOX folder but sometimes you may want to search in both Sent messages and Inbox. In order to do so you have to do give the following command:

imap_conn.select("[Gmail]/All Mail")

Fetching an email without setting it as SEEN

When you want to fetch an email in order to parse you usually do:

typ, msg_data = imap_conn.fetch(uid, 'RFC822')

which will do exactly what you want, but if the email is not read (SEEN) it will mark it as read which something you may not want if you are not building a Gmail client. If you simply want to parse/read the email then do the following

typ, msg_data = imap_conn.fetch(uid, '(BODY.PEEK[HEADER])')

Find the body message and decode it

This can be really tricky as there can be a lot of forwards and replies plus an awful lot of encodings. Usually we the encoding of an email is appended in the subject. The following gist can be to some help:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def _process_body(pl):
	"""find the final payload (body) of an email"""
	if type(pl)==StringType:
		return pl.replace('\r\n>',' ').replace('\r\n',' ').replace('\n\n',' ')
	elif type(pl)==list:
		return _process_body(pl[0])
	elif type(pl)==InstanceType:
		return _process_body(pl._payload)
	else:
		print type(pl)

def _decode_body(subject,body):
	"""decode body if in base64"""
	try:		
		if re.search("(ISO-\d+-\d)", subject):
			iso = re.search("(ISO-\d+-\d)", subject).group(0)
			body = base64.b64decode(body).decode(iso)
			body = text.replace('\r\n>',' ').replace('\r\n',' ').replace('\n\n',' ')			
		return body.encode('utf-8') if type(body)==unicode else body
	except:
		return body.encode('utf-8') if type(body)==unicode else body

If you have any suggestions, corrections please feel free to fork the gists and let me know :)

Permalink | Leave a comment  »

If you are running a web2py app on AppEngine it is matter of sanity to have AppStats logging your app 's performance. It is common practice to have performance monitor for all major web applications out there so that you know where to optimize your application. 

If you are on AppEngine the performance of your application is even more crucial as better performance means less costs and stability (as it is easy to hit the restrictions). 

If you have a web2py application it is very easy to this as web2py is first class wsgi citizen. Simply go to gaehander.py and modify the main function as in the following gist:

1 2 3 4 5

# gaehandler.py line:85 def main():     """Run the wsgi app"""     from appengine_config import webapp_add_wsgi_middleware     wsgiref.handlers.CGIHandler().run(webapp_add_wsgi_middleware(wsgiapp))

Then go to app.yaml and right after the handler definitions add these 2 lines:

1 2 3

handlers: - url: /stats.*   script: $PYTHON_LIB/google/appengine/ext/appstats/ui.py

In the root directory of web2py create a new a file called appengine_config.py with the following content (thanx WritersEar for pointing out):

1 2 3 4

def webapp_add_wsgi_middleware(app):     from google.appengine.ext.appstats import recording     app = recording.appstats_wsgi_middleware(app)     return app

 

And you are good to go!

Permalink | Leave a comment  »

We want SocialCaddy to send HTML emails with daily digests to our users. Well there is nothing fancy about it but we spent one day trying to do that on Google App Engine.

We use the Python framework web2py. After digging with the web2py source code we found a tiny bug that prevented us from sending HTML emails from App Engine. If you face a similar problem then just go to gluon/tools.py and replace line 437 with the following code:

result = mail.send_mail(sender=self.settings.sender, to=to, subject=subject, body=text, html=html)

We hope that no one else will spend 5 hours on such a tiny bug :)

Permalink | Leave a comment  »