Sunday, June 20, 2010

How to download / upload data to Google App Engine for localhost

The appcfg.py --dump option is not working
with this error appcfg.py: error: no such option: --dump

Deprecated method (Code for Configuration) is used instead to upload_data / download_data to localhost app

Here is an example

(1) modify app.yaml and add this under handlers:
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin


You have to restart the app after modifying app.yaml

(2) create this file : album_load.py

# album_loader.py
import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class Album(db.Model):
artist = db.StringProperty()
title = db.StringProperty()
publication_date = db.DateProperty()
length_in_minutes = db.IntegerProperty()

class AlbumLoader(bulkloader.Loader):
def __init__(self):
bulkloader.Loader.__init__(self, 'Album',
[('title', str),
('artist', str),
('publication_date',
lambda x: datetime.datetime.strptime(x, '%m/%d/%Y').date()),
('length_in_minutes', int),
])

loaders = [AlbumLoader]


(3) prepare an upload file : album.csv

title1,artist1,1/1/2000,12
title2,artist2,1/1/2001,22
title3,artist3,1/1/2002,32


(3) Launch your app in localhost and upload using this command

appcfg.py upload_data --config_file=album_loader.py --application=yourappid --kind=Album --url=http://localhost:8080/remote_api --filename=album.csv


(4) check the table data in your Google App SDK console

(5) to download data create this file : album_exporter.py

# album_exporter.py
import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class Album(db.Model):
artist = db.StringProperty()
title = db.StringProperty()
publication_date = db.DateProperty()
length_in_minutes = db.IntegerProperty()

class AlbumExporter(bulkloader.Exporter):
def __init__(self):
bulkloader.Exporter.__init__(self, 'Album',
[('title', str, None),
('artist', str, None),
('publication_date', str, None),
('length_in_minutes', int, None),
])

exporters = [AlbumExporter]


(6) use this command to download data
appcfg.py download_data --config_file=album_exporter.py --application=yourappid --kind=Album --url=http://localhost:8080/remote_api --filename=albumout.csv
.
.
.

2 comments:

alex said...

I've tried your directions but I receive an 'Authentication Failed' error. Any idea what I could be doing wrong?

Anonymous said...

Thanks, very helpful. BTW: I had to do this first before it would work for me:

cd __to_app_directory__
export PYTHONPATH=`pwd`

Alex: Did you put an entry in app.yaml requiring login? This is typically done like so (in app.yaml)

- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin


.. make sure to put this __before__ any wildcard urls that might mask the entry.

.. then you should be prompted for login and pw. Any credentials should work on localhost but I did notice I'd get some rudimentary checking so I entered a valid looking email address and some kind of passwd.