6 replies to this topic

[OrangeWeapons]

Hello, my account name is OrangeWeapons. I'm a newbie here, trying to work with Python and Neopets. I'm using the latest version of Python and urllib.

I'm having issues with the following code, and no knowledge of how to resolve my issues:

import urllib
import http.cookiejar

username = input("Username: ")
password = input("Password: ")

cookiejar = http.cookiejar.LWPCookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar))
opener.addheaders = [("User-agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30")]

form = { "username" : username, "password" : password }
encoded_form = urllib.parse.urlencode(form)
request = urllib.request.urlopen("http://neopets.com/login/", encoded_form)
webpage = opener.open(request)
contents = page.read()

if("logout" in contents):
    print("Logged In!")
else:
    print("Login Failed")

Traceback (most recent call last):
File "C:/Users/John/Desktop/Python/Pet", line 13, in <module>
request = urllib.request.urlopen("http://neopets.com/login/", encoded_form)
File "C:\Python32\lib\urllib\request.py", line 138, in urlopen
return opener.open(url, data, timeout)
File "C:\Python32\lib\urllib\request.py", line 364, in open
req = meth(req)
File "C:\Python32\lib\urllib\request.py", line 1052, in do_request_
raise TypeError("POST data should be bytes"
TypeError: POST data should be bytes or an iterable of bytes. It cannot be str.

Thank-you for your help!

Edited by OrangeWeapons, 22 June 2011 - 05:41 PM.

[Melchoire]

You might hate my solution cause I'm basically telling you to use a premade http client.

But with ruby if I need to do http programming I use mechanize. Cool part is it's also usable in python, you just have to install mechanize...

This code will get a webpage's source would be something like this I think:

#!/usr/bin/python

import mechanize

agent = mechanize.Browser()

res = agent.urlopen("http://www.neocodex.us/")

print res.read()

Edit: it's also full of really cool features, it'll follow redirects, save cookies, give you an array with all the links etc..

[OrangeWeapons]

You might hate my solution cause I'm basically telling you to use a premade http client.

But with ruby if I need to do http programming I use mechanize. Cool part is it's also usable in python, you just have to install mechanize...

This code will get a webpage's source would be something like this I think:

#!/usr/bin/python

import mechanize

agent = mechanize.Browser()

res = agent.urlopen("http://www.neocodex.us/")

print res.read()

Edit: it's also full of really cool features, it'll follow redirects, save cookies, give you an array with all the links etc..

I guess I'll check it out. That is, if I can figure out how to install mechanize on Windows 7

Edited by OrangeWeapons, 22 June 2011 - 07:31 PM.

[Melchoire]

I guess I'll check it out. That is, if I can figure out how to install mechanize on Windows 7

If you wanna stick with your current method, maybe this snippet will help:

import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()

from http://www.voidspace...s/urllib2.shtml

[artificial]

To fix that particular error, change:

encoded_form = urllib.parse.urlencode(form)

To:

encoded_form = urllib.parse.urlencode(form).encode('utf8')

You may also find the following page interest http://docs.python.o...ary/urllib.html

Also note that the urllib.urlopen() function has been removed in Python 3.0 in favor of urllib2.urlopen()

If you want to stick with urllib rather than urllib2, try using Request instead of urlopen. Change:

request = urllib.request.urlopen("http://neopets.com/login/", encoded_form)

To:

request = urllib.request.Request("http://neopets.com/login/", encoded_form)

Also, change the variable on the following line (webpage) to page, since you try to use page.read() without previously declaring it.

One final thing - you're constructing the opener invalidly.
Change:

opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar))

To:

opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookiejar))

HTTPCookieProcessor takes an instance of the cookiejar module rather than the module itself

Works for me after making those changes.

edit: oh, and prior to treating the contents variable as a string, you'll first have to convert it to one. Change:

contents = webpage.read()

To:

contents = str(webpage.read())

Edited by Artificial, 23 June 2011 - 02:24 AM.

[ILOVEPIE]

Hello, my account name is OrangeWeapons. I'm a newbie here, trying to work with Python and Neopets. I'm using the latest version of Python and urllib.

I'm having issues with the following code, and no knowledge of how to resolve my issues:

import urllib
import http.cookiejar

username = input("Username: ")
password = input("Password: ")

cookiejar = http.cookiejar.LWPCookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(http.cookiejar))
opener.addheaders = [("User-agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30")]

form = { "username" : username, "password" : password }
encoded_form = urllib.parse.urlencode(form)
request = urllib.request.urlopen("http://neopets.com/login/", encoded_form)
webpage = opener.open(request)
contents = page.read()

if("logout" in contents):
    print("Logged In!")
else:
    print("Login Failed")

Traceback (most recent call last):
File "C:/Users/John/Desktop/Python/Pet", line 13, in <module>
request = urllib.request.urlopen("http://neopets.com/login/", encoded_form)
File "C:\Python32\lib\urllib\request.py", line 138, in urlopen
return opener.open(url, data, timeout)
File "C:\Python32\lib\urllib\request.py", line 364, in open
req = meth(req)
File "C:\Python32\lib\urllib\request.py", line 1052, in do_request_
raise TypeError("POST data should be bytes"
TypeError: POST data should be bytes or an iterable of bytes. It cannot be str.

Thank-you for your help!

the reason why the top code doesn't work is it's using the GET type of form submission not the POST type which is the one neopets uses.
GET looks like this
http://www.neopets.c...oo&password=bar
whereas POST sends it in the request header, not in the url.

Urlencode means GET not POST.

[artificial]

the reason why the top code doesn't work is it's using the GET type of form submission not the POST type which is the one neopets uses.
GET looks like this
http://www.neopets.c...oo&password=bar
whereas POST sends it in the request header, not in the url.

Urlencode means GET not POST.

I'm sorry, but that's wrong. He's sending post data through, and as far as I'm aware in Python you should always encode it so the HTTP header sent to the server is correctly formatted.

Unfortunately, in Python 3 when you call the urlencode function from the urllib module, it converts the dictionary to a urlencoded string, which the Request function won't accept. You therefore have to convert it back to the correct format, which you can do with the encode function (and it also takes care of any pesky characters that may get in the way). Proof:

from urllib.parse import urlencode

form = { "username" : "test123", "password" : "password123" }

print( "Prior to calling urlencode, type: " + str( type( form ) ) )

encoded_form = urlencode(form)
print( "After calling urlencode type: ", str( type( encoded_form ) ) )

encoded_form = encoded_form.encode('utf8')
print( "After calling utf8 encode type: " + str( type( encoded_form ) ) )

Output:

Prior to calling urlencode, type: <class 'dict'>
After calling urlencode type:  <class 'str'>
After calling utf8 encode type: <class 'bytes'>