Talking to IMAP services can be a source of frustration especially when each service implements parts of the protocol or add some secret ingredients to the sauce. One of the most popular services, as we all know is Gmail.

XOAuth

Gmail adds a lot of nice touches to the mail business. One of the latest additions is the use of XOAuth in order to authenticate against the service without any need of username and password. This can be especially handy if your app your app should read or just parse the headers of the users' emails but you do not want to save passwords, have extra UI for this purpose or let the user revoke the access at will. You can easily use xoauth in your webapp today using this small library.

Example code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def connect_to_gmail(CREDENTIALS, email, oauth_token, oauth_token_secret):
    """
    Call this function to get an authenticated IMAP connection
    """
    consumer = OAuthEntity(CREDENTIALS[0], CREDENTIALS[1])
    access_token = OAuthEntity(oauth_token, oauth_token_secret)
    xoauth_string = GenerateXOauthString(
      consumer, access_token, email, 'imap',
      None, str(random.randrange(2**64 - 1)), str(int(time.time())))

    # connect to imap
    imap_conn = imaplib.IMAP4_SSL('imap.googlemail.com')
    #imap_conn.debug = 3
    imap_conn.authenticate('XOAUTH', lambda x: xoauth_string)
    imap_conn.select('INBOX')

Searching

By default when you use imap.select() you will be able to search within the INBOX folder but sometimes you may want to search in both Sent messages and Inbox. In order to do so you have to do give the following command:

imap_conn.select("[Gmail]/All Mail")

Fetching an email without setting it as SEEN

When you want to fetch an email in order to parse you usually do:

typ, msg_data = imap_conn.fetch(uid, 'RFC822')

which will do exactly what you want, but if the email is not read (SEEN) it will mark it as read which something you may not want if you are not building a Gmail client. If you simply want to parse/read the email then do the following

typ, msg_data = imap_conn.fetch(uid, '(BODY.PEEK[HEADER])')

Find the body message and decode it

This can be really tricky as there can be a lot of forwards and replies plus an awful lot of encodings. Usually we the encoding of an email is appended in the subject. The following gist can be to some help:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def _process_body(pl):
	"""find the final payload (body) of an email"""
	if type(pl)==StringType:
		return pl.replace('\r\n>',' ').replace('\r\n',' ').replace('\n\n',' ')
	elif type(pl)==list:
		return _process_body(pl[0])
	elif type(pl)==InstanceType:
		return _process_body(pl._payload)
	else:
		print type(pl)

def _decode_body(subject,body):
	"""decode body if in base64"""
	try:		
		if re.search("(ISO-\d+-\d)", subject):
			iso = re.search("(ISO-\d+-\d)", subject).group(0)
			body = base64.b64decode(body).decode(iso)
			body = text.replace('\r\n>',' ').replace('\r\n',' ').replace('\n\n',' ')			
		return body.encode('utf-8') if type(body)==unicode else body
	except:
		return body.encode('utf-8') if type(body)==unicode else body

If you have any suggestions, corrections please feel free to fork the gists and let me know :)

Permalink | Leave a comment  »

If you are running a web2py app on AppEngine it is matter of sanity to have AppStats logging your app 's performance. It is common practice to have performance monitor for all major web applications out there so that you know where to optimize your application. 

If you are on AppEngine the performance of your application is even more crucial as better performance means less costs and stability (as it is easy to hit the restrictions). 

If you have a web2py application it is very easy to this as web2py is first class wsgi citizen. Simply go to gaehander.py and modify the main function as in the following gist:

1 2 3 4 5

# gaehandler.py line:85 def main():     """Run the wsgi app"""     from appengine_config import webapp_add_wsgi_middleware     wsgiref.handlers.CGIHandler().run(webapp_add_wsgi_middleware(wsgiapp))

Then go to app.yaml and right after the handler definitions add these 2 lines:

1 2 3

handlers: - url: /stats.*   script: $PYTHON_LIB/google/appengine/ext/appstats/ui.py

In the root directory of web2py create a new a file called appengine_config.py with the following content (thanx WritersEar for pointing out):

1 2 3 4

def webapp_add_wsgi_middleware(app):     from google.appengine.ext.appstats import recording     app = recording.appstats_wsgi_middleware(app)     return app

 

And you are good to go!

Permalink | Leave a comment  »

We want SocialCaddy to send HTML emails with daily digests to our users. Well there is nothing fancy about it but we spent one day trying to do that on Google App Engine.

We use the Python framework web2py. After digging with the web2py source code we found a tiny bug that prevented us from sending HTML emails from App Engine. If you face a similar problem then just go to gluon/tools.py and replace line 437 with the following code:

result = mail.send_mail(sender=self.settings.sender, to=to, subject=subject, body=text, html=html)

We hope that no one else will spend 5 hours on such a tiny bug :)

Permalink | Leave a comment  »

The Facebook platform is the cornerstone of SocialCaddy. As most developers we tend to nag about its APIs and especially the sparse documenation. This time it was no exception. But what has happened?

A few weeks ago ago Facebook announced the Graph. Basically the whole architecture of Facebook is now based on the Graph. Whatever you like on Facebook or beyond it is a node on the Graph and you connect to it. The Open Graph protocol bring the Semantic Web aka Web 3 to our browsers today.

Besides the Graph API Facebook introduced a new permissions policy, new authentication methods (OAuth and a new JS SDK) and it dropped support for some older functionality but without actually announcing it.

First of all we found that you have to upgrade your permissions and if you use Facebook Connect you have to migrate to the new dialog. Go to the Developer app -> Edit Settings -> Migrations -> New Data Permissions.

That was the first step! Remember to study the new permissions.

Now let 's get to the point. Before the Graph we used to select our users' friends work history, education history, activities, etc in a single FQL call like this:

SELECT work_history, education_history, activities, books, movies FROM user WHERE uid IN(some_ids)

Then we updates some of those fields... and... there were no updates! We thought it was some kind of caching but the relationship_status for example was updated! 

Then we decided to give a look at the FQL User table and a lot of the fields were missing but the FQL query was executed but fetching the old values! We could fetch the work_history and education_history doing a simple Graph API call ex.

http://graph.facebook.com/me (or some uid)

That call fetched the missing fields. In the doc page we read that some fields were missing for example relationship_status but it was there in the JSON we got from the Graph API call.

OK, that worked but how could we get all our users' friends work histories in a single call? It seemed we had to query the Graph n times where n is the number of a user 's friends count. That was terrible! Our FQL did not fetch the data and the docs had no sign of the fields we used to ask for. We were desperate but after some careful reading behind the lines our saw that: 

any of its fields which are also fields in the corresponding Graph API version

What does this mean? It means that our old FQL query is still valid (partially) in this format:

SELECT work, education FROM user WHERE uid IN(some_ids)

That made our day! The fields were not lost and in a single request we got all the data we wanted! But what about the books, activities, movies and so on? Well these data is no longer a text field in the user 's profile. They are now connections on the graph so whatever goes there is actually a page you like! Do you want to get all of those in FQL? Just do this

SELECT name,type FROM page WHERE page_id IN (SELECT target_id  FROM connection WHERE source_id=657809015 and target_type="Page")

This will return all the pages your user likes and of course you can filter by type such as MUSICIAN, LOCAL_BUSINESS and so on. The Graph version is much simpler but FQL gives you the power for batch select, advanced filtering and so on.

In conclusion do not think of the Graph API as something different to FQL. Basically they will bring you the same data in different format and using a different way to ask for them. FQL is indeed, as Facebook says, advanced API but it gives you the power to scale more easily and filter your data.

Do you have any similar experiences with the Facebook APIs? Please share your story with us.

Panos

Permalink | Leave a comment  »